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ABSTRACT 


This research examines the usable knowledge gained or refined through 
Distributed Learning from Naval Postgraduate School for personnel serving in 
operational billets. The population studied was students currently enrolled or students 
that had completed the Information Systems and Operations certificate program. The 
study used a web based survey for data collection and used that data to answer four 
research questions. The data clearly demonstrates that Distributed Learning is equivalent 
to resident coursework in terms of usable knowledge gained or refined and distributes 
that usable knowledge quickly and efficiently to individual servicemembers. Beyond the 
individual, a direct transfer to the organization of a portion of the knowledge gained or 
refined occurs. The data suggest that frequency of use of skills can be used as a measure 
of effectiveness for the Distributed Learning program at Naval Postgraduate School, but 
more data are needed to be conclusive. Additionally, the results provide evidence that 
Distributed Learning provides a strong, social interaction learning context. This evidence 
challenges the assertion that social interaction between students and faculty is primarily a 
characteristic of resident coursework and a limitation of Distributed Learning. 
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I. 


INTRODUCTION 


The current trend in manpower management and systems development in the 
operating forces is towards exploiting emerging technologies to minimize redundant 
effort and streamline processes with the intended result of maximizing efficiency. More 
efficient systems and processes move towards the desired state of employing the bulk of 
resources available to war-fighting competencies or direct support of war-fighting 
competencies and reducing requirements of competing efforts for those resources in 
support roles. Harnessing emerging technologies and applying them thoughtfully 
provides more precision and economy of force with less dedicated resources, both in 
manpower and systems. This provides the operating forces with increased flexibility as 
world-wide commitments and contingencies continue to grow. Implementation of new 
technologies requires resident knowledge in order to succeed. While contractor support 
is invaluable and necessary, it is a finite resource and does not replace the need for that 
knowledge as an organic asset. Graduate-level education is one means to develop and 
maintain this organic knowledge base. The United States Navy (Navy) promotes 
educational opportunities to include graduate education for its members primarily to 
develop and maintain this base. A central concern for the Navy is the question of how to 
obligate finite resources to maximize the benefits of graduate education for the 
organization. [Filizetti 2003]. 

The question of how to best allocate financial and manpower resources to 
maximize graduate education benefits begins with an analysis of available methods to 
provide that education. For both the Navy and the United States Marine Corps (USMC), 
the primary means of attaining the organizational resource of organic graduate-trained 
personnel is resident course work at the Naval Postgraduate School (NPS). “The mission 
of the Naval Postgraduate School is to provide relevant and unique advanced education 
and research programs in order to increase the combat effectiveness of U.S. and Allied 
armed forces and enhance the security of the United States.” [NPS 2005]. Resident 
coursework, no matter how effective, has some limitations in terms of application in the 
Fleet. 
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First, a service loses the use of an officer for up to twenty- seven months while 
that officer is attending NPS. Second, graduates may not report to billets that apply their 
new skill sets directly to the tactical concerns of the operating forces. Distributed 
Learning (DL) at NPS is a means available to the services that address the limitations of 
resident coursework at NPS. DL is administered at NPS by the Office of Continuous 
Learning (OCL). “The mission of the OCL is to coordinate and administer innovative and 
cost-effective efforts to identify, package and deliver the intellectual capital of NPS to the 
Navy and Department of Defense (DoD) decision-makers and other component personnel 
who are not able to attend NPS on a full- time residential basis” [NPS 2005]. 

Accepting DL coursework as a viable alternative to resident study at NPS for the 
individual services requires qualitative and quantitative measures that demonstrate NPS’s 
ability to provide needed graduate-level skills to personnel in the operating forces. 
Without a demonstrated measure of usable knowledge gained, the flexibility and 
convenience of DL coursework from NPS become the primary incentive for an 
organization to sponsor an officer’s study. In effect, the organization then loses the 
benefit of the social interaction and development recognized as a valuable but indefinable 
aspect of resident study that helps students “convert their tacit knowledge into explicit 
knowledge” [Na Ubon 2002], while at the same time potentially not gaining a specific, 
tangible learning outcome that targets needed skills for that organization. Essentially, the 
organization is in a position that requires commitment of resources with no specified or 
expected return. This need to develop a measure for the successful transfer of usable 
knowledge via DL is the genesis of this research. 

Measuring the specific value of DL coursework for personnel and organizations 
in the operating forces begins with the basic question of whether DL coursework is 
comparable to resident study. Acknowledging the challenges in time and space 
associated with any DL program, comparing learning outcomes between a resident and 
DL class for the same course provides insight into this basic question. In fact, academic 
evidence exists that learning outcomes between resident and non-resident work can be 
comparable [Johnson 2000]. A blind study review conducted at the University of Illinois 
at Urbana-Champaign reported no significant statistical difference between final project 
evaluations between a resident and DL group of students taking the same course [Johnson 
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2000]. Assuming equivalence between resident and DL coursework, the primary 
question becomes what parameter is best suited to measure DL effectiveness for the 
operating forces. 

Understanding student preference impact on survey results and potential 
organizational bias from resource allocation become important considerations when 
determining a concrete measure of usable knowledge gained via DL work. Comparing 
learning objectives for a DL course to the actual learning outcomes used in performance 
of billet requirements in the operating forces is the key to isolating a quantitative measure 
of effectiveness. Specifically, frequency of use of learning outcomes gained through DL 
work provides a potential means and measure of NPS DL success in providing timely, 
usable knowledge to the forward edge of the operating forces. Frequency of use of 
specific skill sets to demonstrate usable knowledge gained in a graduate environment has 
a research precedent for resident study at NPS [Branstetter 2002]. 

Working from the position that graduate skills at a Master’s level is finding more 
and more application in the operating forces as technology advances and that resident 
coursework at NPS does have some drawbacks for organizations in the operating forces, 
NPS’s DL programs provide a potential means for effectively meeting the need for 
graduate skill sets in the Fleet billets. By measuring the effectiveness of DL in terms of 
usable knowledge gained, receiving organizations as well as the NPS DL program 
benefit. The former gets a way to measure return on investment or at least an expectation 
of results. The latter receives input that aids tailoring individual programs to maximize 
results for the operating forces. Interestingly, DL students learning a specific skill set that 
they are simultaneously reinforcing in billet performance may actually maintain a higher 
level of retention than students that gain the same skills in a strictly academic setting 
[Conway 1991]. Providing a method to assess the effectiveness of DL coursework 
within the operating forces in terms of usable knowledge gained is the ultimate aim of 
this research and intended to help the NPS DL program provide the most responsive 
product possible to personnel serving in the operating forces. 
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II. KNOWLEDGE CREATION 


In the Information Age, managing knowledge better than rivals provides a distinct 
competitive advantage for an organization regardless of the specific field or endeavor in 
this day and age. The Armed Forces are no different. Management of knowledge in the 
ever-changing technological environment at the tactical level coupled with the 
requirement for mission success demands a concentrated effort to maximize the 
knowledge within the operating forces. A critical aspect of knowledge management in 
the operating forces is an acknowledgement of the nature of the work force. Generally, 
individuals are trained to perform specific, highly technical tasks without an 
understanding of the larger picture of how the other systems and sub-systems interrelate 
to meet the larger mission. In this situation, where multiple processes that are 
implemented individually but designed to provide a synergistic result, the organization 
must contain a sufficient level of knowledge to manage the integration of the separate 
efforts in order to be successful. While organizational knowledge contains many aspects, 
the basic building block of organizational knowledge is creation of new knowledge. 

A. ORGANIZATIONAL KNOWLEDGE CREATION THEORY 

Based upon the quantity and logical progression of the research conducted in this 
area by Ikujiro Nonaka, Ph.D. and the quality of his research as evidenced by the heavy 
citation of his work in other published research in this area, the Nonaka-based models’ 
progression form the base of this research’s literature review. This literature review uses 
Nonaka-based organizational knowledge creation theory to compare and contrast other 
knowledge creation models with the intent of either reinforcing or rejecting elements of 
that theory within the perspective of this research effort. Accepting that knowledge is 
categorized as either tacit or explicit and that the summation of both held by an 
organization represents the total organic knowledge “owned” by that organization leads 
to an expectation that any model of knowledge creation requires addressing both 
categories. In fact, the interaction between these two categories of knowledge is the key 
ingredient in creating “new knowledge” internal to an organization [Nonaka 1994]. 
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1. Four Modes of Knowledge Interaction Model 

The Nonaka and Takeuchi model of knowledge creation considers four modes of 
interaction between explicit and tacit knowledge, which results in the creation of new 
knowledge. These four modes of interaction are described as follows: (1) tacit to tacit, 
(2) tacit to explicit, (3) explicit to tacit, (4) explicit to explicit. These modes of 
interaction are further developed through the means of interaction that results in the 
knowledge transfer and/or creation. Tacit to tacit knowledge creation is achieved through 
a process of socialization, which results from a transfer of experienced-based knowledge 
between individuals in an environment that reinforces that knowledge transfer such as on- 
the-job-training [Nonaka 1994]. Explicit to explicit knowledge creation occurs through a 
combination process, which results from an interaction of individuals in a collaborative 
environment where batches of explicitly held knowledge are combined to enhance 
organization knowledge and understanding in an area [Nonaka 1994]. Tacit to explicit 
knowledge creation occurs through an extemalization process that is simply described as 
when tacit knowledge is codified in a manner where other individuals can access it for 
use without having to gain the knowledge though experience [Nonaka 1994]. Explicit to 
tacit knowledge creation occurs through an internalization process where an individual 
receives knowledge in explicit form and internalizes or learns that knowledge sufficiently 
to enhance his or her level of tacit knowledge in that area [Nonaka 1994]. 

Because internalization of knowledge is the mode of knowledge conversion that 
best correlates with individual learning through a DL format in relative terms to the focus 
of this research, it is the mode of the Nonaka and Takeuchi model that is of the most 
interest for this research effort. Internalization is more than just the conversion of explicit 
knowledge into tacit knowledge in one individual, knowledge that is internalized by an 
individual into their tacit knowledge base becomes an asset for the organization through 
such assets as shared mental models or technical know-how [Nonaka 2000]. 

Establishing that internalization of new knowledge occurs within an organization 
validates the potential that the newly acquired knowledge can initiate a new knowledge 
spiral within the organization. The Socialization, Extemalization, Combination, and 
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Internalization (SECI) model depicted in Figure 1 contains a visual description of the 
interaction and relationships between the four modes of knowledge interaction [Nonaka 
1994], 


Tacit 

knowledge 


From 


Explicit 

knowledge 


Tacit knowledge T Explicit knowledge 


Socialization 

External ization 

Internalization 

Combination 


Figure 1. SECI Model (From Nonaka 1994) 

a. Characteristics of Knowledge Transfers Between Explicit and 
Tacit Knowledge 

Developing the assertion that knowledge conversion as explained by the 
SECI model occurs because of knowledge transfers between tacit and explicit knowledge 
bases in various modes presents the necessity to examine the ease of knowledge flow 
between those two types of knowledge. If the flow from explicit to tacit is relatively easy 
to recognize and capture as compared to the flow of tacit to explicit or vice versa, then 
that becomes a consideration in attempting to manage the knowledge creation process. If 
a disparity in the ease of knowledge flow based solely on the direction of that flow exists, 
then by extension some modes of knowledge conversion in the SECI model become more 
difficult to complete than others. Furthering this point is the intuitive consideration that a 
tacit to explicit knowledge flow is more difficult to enable than an explicit to tacit 
knowledge flow just because of the nature of tacit knowledge. Figure 2 depicts Inkpen 
and Dinur’s assertion that as knowledge tacitness increases its ease of transferability 
decreases, especially as the scope of that transfer increases as well [Inkpen 1998]. 
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Individual ^ Group ► Organization 

Low Knowledge 
Tacitness 


♦ 

High Knowledge 
Tacitness 


High Ease of Transferability 
Low Complexity 

_ - 


X 

X 

Low Ease of Transferability 
High Complexity 


Figure 2. Knowledge Transfer Classification Framework (From Inkpen 1998) 

b. Organizational Characteristics’ Effect on Knowledge Transfer 
Continuing to provide more amplification and theoretical support of the 
two types of knowledge and their interaction involves in terms of organizational 
knowledge creation requires some examination of the characteristics of organizations. 
Specifically, knowledge transfers differ based upon factors such as size and perspective 
of the receiving element of a knowledge transfer. Figure 2 alludes to this element of 
knowledge transfer in that it visually connects ease of transferability, complexity, and 
tacitiness of knowledge with individuals, groups, and organizations. Furthering this 
observation, Gunnar Hedlund recognizes four “agents of knowledge”, which are related 
to the specific sub-sets of organizational relationships. He asserts that tacit and explicit 
knowledge, which he refers to as articulated knowledge, exists across all of the “agents of 
knowledge” and provides specific examples in Figure 3 [Hedlund 1994]. 
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Figure 3. Agents of Knowledge (From Hedlund 1994) 


While practical application of knowledge transfers may be impacted by 
the direction of knowledge flows and organizational characteristics, the SECI model 
serves as a theoretical model to continue examination of the knowledge creation process. 
Nonaka and Takeuchi further develop their model of the interaction between the types of 
knowledge to represent the way new knowledge is created in an organization by these 
interactions. They describe the continuous interaction between tacit and explicit 
knowledge as a “spiral” that works to build new knowledge from existing knowledge. 
“While each of the four modes of knowledge creation can create new knowledge 
independently, the central theme of the model of organizational knowledge creation 
proposed here hinges on the dynamic interaction between the different modes of 
knowledge creation. That is to say, “knowledge creations centers on the building of tacit 
and explicit knowledge and, more importantly, on the interchange between those two 
aspects of knowledge through internalization and extemalization” [Nonaka 1994]. 

2. Nonaka’s Knowledge Spiral 

The interaction between tacit and explicit knowledge providing the catalyst for 
new knowledge creation is supported by general observation as a supporting mechanism 
for the formal research in the area. In a generic military application, which is the 
intended environment of this research, a work section will have standard operating 
procedures (SOPs). When an individual arrives in a billet in that section, written 
procedures, SOPs, and written guidance provides a general understanding of the billet 
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responsibilities and billet procedures. That written direction, explicit in nature, coupled 
with the experience of his or her co-workers, aid in building a level of tacit knowledge. 
Over time, that individual becomes the subject matter expert and develops new 
techniques, tactics, and procedures (TTPs) for processes in his or her work section. Once 
those new TTPs are codified in written procedures and SOPs, that tacit knowledge, which 
was built upon existing explicit knowledge and nurtured by other individual’s 
experiences (tacit knowledge), has become explicit in nature. That entire process occurs 
again and again over time and is an example of how the “knowledge spiral”, i.e., the 
interaction between tacit and explicit knowledge through the externalization and 
internalization, builds new knowledge in an organization as depicted in Figure 4 [Nonaka 
1994]. This knowledge spiral encompasses all four of the knowledge conversion modes 
as outlined in the SECI and demonstrates a relationship between the two types of 
knowledge; individuals and organizations; and the philosophical and metaphysical 
dimensions. 
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Figure 4. Knowledge Spiral (Nonaka 1994) 
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B. ORGANIZATIONAL KNOWLEDGE CONVERSION THEORY 


While the knowledge spiral models the knowledge creations aspects of the 
interaction between tacit and explicit knowledge, it does not specifically address the 
environment and related intangible factors that influence the knowledge creation process. 
Specifically, the knowledge spiral describes the continuous exchange between tacit and 
explicit knowledge in the context of knowledge creation in an organization but does not 
directly model the crucial, contributing organizational elements of the process. Because 
knowledge creation in an organization is most certainly a dynamic process, a more robust 
model that captures those elements is necessary. Nonaka, along with Toyama and 
Konno, build upon Nonaka’s previous work and propose a model that captures more of 
the contributing factors associated with the knowledge creation aspects of the knowledge 
spiral. Their proposed model is presented in Figure 5 [Nonaka 2000]. 


B y Co nt ext-Knowledge Plac e 


•Platform for knowledge 
conversion 

•Space for self-transcendence 
•Multi-context place 


Quality and 
Energy 
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Figure 5. 


Organizational Knowledge Conversion Process (Nonaka 2000) 


“This model consists of three elements: the SECI process, (i) knowledge creation 
through the conversion of tacit and explicit knowledge; (ii) ba the shared context for 
knowledge creation; and (iii) knowledge assets, the inputs, outputs and moderators of the 
knowledge-creation process.” [Nonaka 2000]. 
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1. Four Modes of Knowledge Interaction Model Including Descriptors 

While the original SECI model presented by Nonaka and Takeuchi, which is 
depicted in Figure 1, demonstrates the knowledge conversion process, Nonaka, Toyama, 
and Konno updated the SECI model in their collective work. Their updated SECI model 
is presented in Figure 6 [Nonaka 2000]. While the basic premise of this model remains 
the same, Nonaka, Toyama, and Konno add descriptors to the conversion modes in the 
model. These descriptors are important additions to the model because they drive to the 
point that Nonaka, Toyama, and Konno pursue in that the three elements associated with 
the knowledge creation process as depicted in the model presented in Figure 6 operate 
interdependently and cannot be truly administered independently. 



Figure 6. SECI Model with Descriptors (From Nonaka 2000) 

Empathizing describes the key element of the socialization mode of conversion 
because it delineates the essential characteristic of knowledge transfer required to transfer 
tacit knowledge to tacit knowledge. “Since tacit knowledge is difficult to formalize and 
often time- and space-specific, tacit knowledge can be acquired only through shared 
experience, such as spending time together or living together in the same environment” 
[Nonaka 2000]. Essentially, an empathic conduit must be built between individuals prior 
to the transfer of tacit knowledge. Articulating is the key descriptor for the externalizing 
mode of knowledge conversion because it describes the single most necessary element 
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involved with a tacit knowledge to explicit knowledge conversion. Converting tacit to 
explicit knowledge requires the holder of the tacit knowledge to crystallize a portion of 
his or her relevant tacit knowledge base into a form that can be expressed and understood 
in an explicit format for the recipient of that knowledge transfer. “The successful 
conversion of tacit knowledge into explicit knowledge depends on the sequential use of 
metaphor, analogy, and model” [Nonaka 2000]. Connecting is the key descriptor for the 
combination mode of knowledge conversion. Converting explicit knowledge to other 
more usable or pertinent explicit knowledge sets requires taking inputs from a myriad of 
sources, potentially both internal and external to the organization, in order to generate 
new explicit knowledge for the organization. Database and network technologies are a 
key enabler of the required connectivity for this type of knowledge conversion [Nonaka 
2000]. Embodying is the key descriptor for the internalization mode of knowledge 
conversion. Converting explicit knowledge to tacit knowledge requires an individual to 
add to his or her tacit knowledge base upon receipt of knowledge distributed in a codified 
format. Explicit knowledge is embodied through “hands-on” work coordinated with 
explicit knowledge inputs as well as simulations or situation specific experiments that 
achieve a successful “learning by doing” methodology [Nonaka 2000]. 

Clearly, this refinement and extension of the knowledge spiral model presented 
earlier by Nonaka and Takeuchi allows for a better understanding of the entire knowledge 
creation effort within an organization. In fact, the SECI process portion of this model, 
which is only one of the three elements included, is basically the foundation for the 
knowledge spiral modeled by Nonaka and Takeuchi. The other two elements are 
concerned with the lateral knowledge collection, transfer and, acceptance within an 
organization and the management of knowledge creation within an organization. 
Understanding the environmental and/or external factors that enhance or hinder 
knowledge creation within an organization is a pivotal point for this research. Although 
the basis of this research is measuring usable knowledge gained, the hope is that follow- 
on research will be able to offer recommendations that provide significant knowledge 
management enhancements in terms of introducing new and usable knowledge into the 
operating forces via DL. Basically, this model acknowledges knowledge creation within 
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an organization, and it also acknowledges that the knowledge creation process can be 
positively or negatively influenced through management. 

2. Shared Context 

While the SECI process relating to the knowledge spiral remains a foundational 
element of this knowledge creation model, this model formally recognizes the 
significance of the environment in which knowledge transfer occurs. The primary 
component of concern with the environment for knowledge transfer is the Nonaka’s ba. 
Ba , a Japanese word that roughly translates to “place”, refers to the context in which the 
knowledge spiral occurs. Nonaka, Toyama, and Konno assert that knowledge is not a 
stand-alone entity but is context-specific relative to the individuals and environment 
interacting with said knowledge [Nonaka 2000]. Figure 7 is a visual representation of 
the knowledge spiral depicted in Figure 4 incorporating individual perspective and 
interaction with knowledge creation within a shared context, or ba [Nonaka 2000]. 



Figure 7. Knowledge Spiral with Shared Context (From Nonaka 2000) 

Further developing the idea of a shared context for knowledge creations pushes 
towards a need for a deeper understanding of the shared context and its effects on 
knowledge creation. 

Previous points in this research have supposed that knowledge is created within 
an individual; however, previous points have also hypothesized that the interaction 
between individuals and their immediate environment, i.e., the four modes of knowledge 
transfer described in the SECI process. In reality, the concept of a shared context and its 
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importance to knowledge creation is just an extension of and a melding of these two 
elements of Nonaka and Takeuchi’s original model. A shared context just realizes that 
the interaction necessary for knowledge transfer and thus knowledge creation occurs 
between individuals that share a time and space that provides them a relatively shared and 
cohesive perspective into the application of knowledge. 

One aspect that has not been addressed is the genesis of the primary research used 
as the basis for this research effort. Namely, Nonaka’s work with various colleagues over 
a span of more than a decade provides the foundation for this knowledge creation 
literature review. Because Nonaka’s work centers on the dynamics of organizational 
knowledge creation in Japanese industry and because many of his colleagues on sources 
cited in this research are of Asian decent as well, a consideration must be made of any 
potential bias of unique perspective on the theory of knowledge creation specifics related 
to cultural identity. In fact, an expectation of a difference between an Eastern and 
Western mindset leads to the assumption that fluctuations exist in the application of 
knowledge creation theory and have to be noted. “Any model which can make sense of 
these differences is a stronger candidate for a more general theory than those limited by 
behavioral and other assumptions peculiar to one or the other nation, tribe, etc.” [Hedlund 
1994], 

Hedlund proposes an organizational knowledge creation model seen in Figure 8 
similar to the Nonaka inspired models, but it is a bit more generalized and specifically 
recognizes the storage, transfer, and transformation of knowledge [Hedlund 1994]. 
Before comparing this model with the Nonaka based models, the terms he uses in his 
model must be defined. Hedlund’s term articulated knowledge is roughly equivalent to 
explicit knowledge as used in the balance of this research. Articulation is tacit 
knowledge converting to explicit knowledge. Internalization is explicit knowledge 
converting to tacit knowledge. Reflection is the interaction between tacit and explicit 
knowledge sets. Extension is the transfer of knowledge, both tacit and explicit, from 
lower levels in the organization to higher levels in the organization. Appropriation is the 
reverse of extension and dialogue is the interaction between the two. Assimilation and 
dissemination are the inputs and outputs of this process [Hedlund 1994]. 
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With terms defined, analysis of the two organizational knowledge creation models 
provides some insight into the possible effects of cultural perspective. Initial observation 
of the models shown in Figure 8 and Figure 3, inclusive of the information in Figure 5 
plus Figure 6, which are considered together, demonstrate a significant amount of 
similarity. Both models key on the interaction between tacit and explicit knowledge 
bases as the foundation for organizational knowledge creation. Both models also address 
knowledge transfer through different levels of the organization. Hedlund’s model uses 
the exact same naming conventions for levels of an organization used in Nonaka’s 
knowledge original spiral, which is depicted in Figure 4. 



Figure 8. Hedlund’s Organizational Knowledge Creation Model (From Hedlund 

1994) 

Interestingly, while Hedlund’s model encapsulates the same essential elements as 
the Nonaka-based theories, it is clearly more linear in design than the Nonaka-based 
models, which is perhaps a bit of a nod to the Western mindset. One of the 
characteristics of the model that support that assertion are the pairing of arrows 
representing various knowledge transfers compared to the circular characteristics of the 
knowledge transfers in Nonaka’s models. A second characteristic of the model that 
demonstrates linear thought is the clear delineation of inputs and outputs into the process. 
Another key difference between the models is the specific capture of knowledge transfers 
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occurring both up and down the hierarchy of an organization. Examination of these 
models details some differences; however, the basics of the organization knowledge 
creation theory of the Nonaka-based models appear in full in both models. Because the 
primary differences in the models do not seem attributable to the underlying theoretical 
concepts, the Nonaka-based theory continues to serve as the foundation for this 
research’s literature review. 

Accepting the premise and importance of a shared context for knowledge creation 
and acknowledgment of possible cultural differences applicable to implementation of a 
knowledge creating process leads to a need to understand how to prepare and facilitate 
that shared context in the most beneficial way in order to encourage and maximize the 
necessary interactions between individuals and environment required by the knowledge 
creation process. The first realization regarding the shared context is that the edges of 
that context form a barrier that is constrained by the specific task, time, current 
organizational competitive situation, and organizational culture. The result of this 
realization is that the shared context involved in the knowledge creation process is fluid 
and changes according to task, individuals involved, and current organization culture 
[Nonaka 2000], 

Because the shared context element of knowledge creation serves as “melting 
pot” of perspectives, it follows that there are multiple means of influencing or 
experiencing this shared context. In fact, Nonaka, Toyama, and Konno identify four 
types of ba: “originating bci. dialoguing ha. systemizing ha. and exercising ha. which are 
defined by two dimensions of interactions (see Figure 9) [Nonaka 2000]. The two 
dimensions considered by Nonaka, Toyama, and Konno are the type of interaction and 
the media through which that interaction takes place. Basically, interactions occur at the 
individual or related group level, and they occur either face-to-face or through a 
secondary means of communication. 
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Face-to-face 

Media 

Virtual 

Figure 9. Types of Interaction (From Nonaka 2000) 

Before a detailed examination of Nonaka’s integration of the ba, shared context, 
into his knowledge creation theory, the underlying concept of how immediate 
environment impacts knowledge transfers in “real world” application needs to be 
examined. Self-reflection upon past duties and tasks in different organizations verify that 
there are “ways of doing business” that vary widely among similar processes depending 
upon the leadership-driven culture of the organization or sub-elements of the 
organization. In military culture, the Commanding Officer sets the command climate, 
which becomes the shared context for his or her staff and subordinate commanders. 
When a change of command occurs, it is not uncommon for the same set of staff and 
subordinate commanders to alter the “way they do business” to fit the new Commander’s 
intent, (i.e., the new shared context for the organization). Usually, personnel in this 
situation will take the best of the old “ways of doing business” and meld it into the 
processes consistent with the new “way of doing business”. Essentially, the organization 
creates new knowledge driven by the change in shared context. This phenomenon leads 
to the desire to identify the theoretical agent responsible for this initiation of change in 
shared context leading to the creation of new knowledge within the organization. 

Using the scenario of a change of command develops a backdrop to review the 
nature of a shared context in a military setting. When a change of command occurs, the 
environment above the changing commander does not change, and as such, there is no 
expectation that the shared context influences on knowledge creation will change 
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significantly either. However, from the changing command on down the hierarchy of the 
command structure, the shared context faces potentially significant change with 
corresponding interactions leading to new knowledge. Furthering this point of inspection 
leads to seeking a connection between organization environment and knowledge creation. 
Matusik defines the organizational environment, or shared context, in terms of 
knowledge. She describes the knowledge held within an organization as private 
knowledge and also as a genesis of competitive advantage [Matusik 1998]. From the 
Matusik model presented in Figure 10 [Matusik 1998], many of the knowledge creation 
theory components previously discussed reappear. The unique feature of this model is 
the delineation between private and public knowledge. Clearly, any influence upon the 
private knowledge of an organization is going to subsequently influence the knowledge 
components of that organization. The link between shared context and knowledge 
creation occurs most vigorously when a command climate change is thrust upon an 
organization because it is during that time of change that robust interactions are forced to 
occur between tacit and explicit knowledge stores and between individuals and sub¬ 
elements of the organization. Inspection of Figure 10 helps visualize the relationship 
between a shared context and knowledge creation within an organization. 

Organizational Knowledge Components 


Private 

(tirm-specific) 



Component 


Architectural 



Public 


V 

Component 




Individual 



Tacit Explicit 


Figure 10. Organizational Knowledge Components (From Matusik 1998) 
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Completing a scenario based inspection of the link between shared context and 
knowledge creation leads to the further inspection of the theoretical components and 
interactions of the shared context, or ba. Just as in the SECI model of knowledge 
conversion modes depicted in Figure 1, understanding the individual types of ba is 
necessary in order to fully appreciate the interaction between the individual types. Again, 
just as in the SECI model, the interaction between the separate components is the agent 
responsible for supporting knowledge creation. In fact, there is a correlation between the 
individual modes of interaction from the SECI model and the individuals types of ba 
presented in Figure 9. This relationship is logical because the knowledge conversions 
depicted in the SECI model each lend themselves to a specific supporting context. 
Originating ba is defined by Nonaka, Toyama, and Konno as individual to individual 
interactions. This type of ba correlates to the socialization mode of knowledge 
conversion because face-to-face communication contains the body language aspects of 
communication that best augment the conveyance of tacit knowledge [Nonaka 2000]. 
Dialoguing ba is defined as the collective and face-to-face interactions. This type of ba 
correlates to the extemalization mode of knowledge conversion. It is characterized by 
back and forth discussion and self-reflection, which is the medium best applicable to 
encourage tacit to explicit knowledge conversion because individually held mental 
models and technical skills are developed into workable concepts [Nonaka 2000]. 
Systemizing ba is defined by interactions that occur through multiple mediums, both 
collective and virtual. This type of ba correlates to the combination mode of knowledge 
conversion, which is essentially the conversion of explicit knowledge into more tightly 
defined sets or sequences of explicit knowledge assets. Systemizing ba supports this 
mode because of it is characterized by explicit knowledge that is easily distributed via 
multiple possible mediums to a large group [Nonaka 2000]. Exercising ba is defined as 
interactions that occur virtually and individually. This type of ba correlates to the 
internalization mode of knowledge conversion because it involves individuals receiving 
explicit knowledge via a virtual medium with the intent of adding to the individual’s tacit 
knowledge base [Nonaka 2000]. 

Figure 11 is a representation of the correlation between each individual type of 
shared context and the mode of knowledge conversion from the SECI model that it best 
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supports. This correlation warrants emphasis because it draws a linear based knowledge 
flow connection between the Nonaka-based models and the Hedlund model, which is 
important to this research because it is primarily concerned with officers in the United 
States Armed Forces. Specifying a specific shared environment’s role with a specific 
knowledge conversion mode develops a partial cause and effect relationship for the 
intended recipients of this research, which possibly provides a sense of familiarity to 
military officers trained in depth in sequenced processes. 


Shared Context (Ba) 


Sunnorts Knowledge Conversion Mode 




Figure 11. Relationships Between Context and Knowledge Conversion 


3. Knowledge Assets 

Understanding the effect of the correlation between a specific shared context and 

a specific mode of knowledge conversion captures a significant portion of the knowledge 

creation process in Nonaka, Toyama, and Konno’s knowledge creation model depicted in 

Figure 6. The remaining element requiring consideration in an organizational knowledge 

creation process is the knowledge assets held by that organization. “Knowledge assets 

are the inputs, outputs, and moderating factors of the knowledge-creation process” 

[Nonaka 2000]. While this definition for knowledge assets is concrete and relatively 

simple conceptually, the application of knowledge assets in the knowledge creation 

process is where the difficulty lies. The difficulty in managing and utilizing knowledge 
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assets emanates from their nature. Knowledge assets are organization-specific. As such, 
understanding and managing those assets demand continual focus within the 
organization, and that management cannot be applied through a generic “book answer” 
type of approach. Additionally, because knowledge assets are organization specific, in a 
military organization those assets are prone to a high level of fluctuation due to the 
transitory nature of personnel in key positions. Beyond the fluidity associated with 
knowledge assets, they are extremely dynamic and somewhat evolutionary in themselves 
because they consist in large part of inputs and outputs of an organization’s knowledge 
creation efforts [Nonaka 2000]. 

Nonaka, Toyama, and Konno identified four categories of knowledge assets. 
Their four categories of knowledge assets are experimental knowledge assets, conceptual 
knowledge assets, systemic knowledge assets, and routine knowledge assets. Not 
surprisingly, the categories of knowledge assets “fit” onto the updated SECI model and 
the types of ba model. This symmetry of models is obvious from the knowledge assets 
categorization model presented by Nonaka, Toyama, and Konno as depicted in Figure 12 
[Nonaka 2000]. 

Prior to working through the concepts of knowledge assets in detail, observation 
of Figure 12 identifies the absence of a specific reference to the “lifelong” tacit base of 
knowledge held by individuals in favor of the individual tacit knowledge base relating to 
specific skill sets and processes. Surely, an individual that has years of experience that 
has been augmented with both formal and informal training and education possesses a 
substantial capability to enhance knowledge flows within an organization, even if specific 
tacit knowledge related to relevant organizational processes is immature. Basically, since 
people are an asset to an organization, the tacit knowledge bases built by those 
individuals prior to joining the organization are knowledge assets the organization can 
exploit and use just by the default of employing that individual. The concept of careers 
as a knowledge repository of all of the skills ever used and developed by and individual 
vice just a sequence of jobs supports the concept of a “lifelong” tacit knowledge base at 
the individual level as a knowledge asset for gaining organizations [Bird 1994]. 
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Experiential Knowledge Assets 

Tacit knowledge shared through 
common experiences 

• Skills and know-how of individuals 

• Care, love, trust, and security 

• Energy, passion, and tension 

Conceptual Knowledge Assets 

Explicit know ledge articulated through 
images, symbols, and language 

• Product concepts 

• Design 

• Brand equity 

Routine Knowledge Assets 

Tacit knowledge routinised and 
embedded in actions and practices 

• Know-how in daily operations 

• Organisational routines 

• Organisational culture 

Systemic Knowledge Assets 

Systemised and packaged explicit 
knowledge 

• Documents, specifications, manuals 

• Database 

• Patents and licenses 


Figure 12. Knowledge Assets Categorization Model (From Nonaka 2000) 

Exploring the knowledge assets as presented in the Nonaka-based model begins 
with experiential knowledge assets. Experiential knowledge assets, the tacit knowledge 
acquired through shared experiences/environment within an organization, span the 
spectrum from emotional to skill specific elements of an organizational tacit knowledge 
base. Because this type of knowledge asset relies upon common experience, it is 
organization specific and can only be developed or acquired through an organization’s 
experiences resulting from pursuit of mission objectives and the processes employed to 
meet those respective mission objectives [Nonaka 2000]. Because an organization’s 
experiential knowledge assets reside in a tacit form, a recognizable association emerges 
with the originating type of shared context and the socialization mode of knowledge 
conversion. 

Conceptual knowledge assets exist in explicit form. Nonaka, Toyama, and Konno 
define these assets as “explicit knowledge articulated through images, symbols, and 
language” [Nonaka 2000]. Because conceptual knowledge assets are explicit in nature, 
an assumption holds that the initial level of understanding of these assets is higher than 
the other forms of knowledge assets. A key characteristic of conceptual knowledge 
assets is that they are not necessarily tightly codified or specifically anchored to a process 
or system, even though they are explicit in nature. Because of their explicit nature and 
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their “high level” characteristic, conceptual knowledge assets seem to have an association 
with the dialoguing type of shared context and the extemalization mode of knowledge 
conversion. 

Systemic knowledge assets consist of formalized explicit knowledge that directly 
supports a process or system. “A characteristic of systemic knowledge assets is that they 
can be transferred easily” [Nonaka 2000]. Systemic knowledge assets differ from the 
explicit knowledge assets that are classified as conceptual knowledge assets in that 
systemic knowledge assets are highly codified and associated with specific processes. 
Recalling that knowledge assets function as inputs and outputs and potentially 
demonstrate an evolutionary nature helps define systemic knowledge assets. For 
example, documentation for a process is a systemic knowledge asset for an organization. 
Using that documentation as an input into the greater knowledge creating processes of an 
organization may result in an output from that greater knowledge creating process of a 
refined or updated process, which then in turn results in an output of refined or updated 
documentation. This example demonstrates the explicit to explicit knowledge conversion 
that occurs in the combination knowledge conversion mode, which is supported by the 
systemizing shared context. 

Routine knowledge assets are exactly what their name suggests. They are tacit 
knowledge assets ingrained into routine process and actions within an organization. 
Because routine knowledge assets are nurtured “on the floor”, they have the distinction of 
being highly specific and workable bits of knowledge. Essentially, routine knowledge 
assets are practical [Nonaka 2000]. Additionally, routine knowledge assets become so 
much a part of the individuals and processes “on the floor” that they can be dismissed as 
“just the way things are done”. However, because routine knowledge is applied at the 
production or tactical application level, it has in many ways the potential to induce the 
most immediate and noticeable results in the knowledge creating endeavors for an 
organization. Because instruction in a daily activity is explicit but exercising that 
instruction taps into the tacit knowledge held in a process and builds the tacit base of the 
individual, routine knowledge assets roughly correlate with the exercising shared context 
and the internalization mode of knowledge conversion. 
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One key element not detailed explicitly in this model is the role of the different 
layers of management in the knowledge creation process; however, the role of 
management in the knowledge creation process is a foundational element for Nonaka’s 
model. Nonaka highlights the responsibilities of management and their critical role in the 
knowledge creation process in context of the SECI knowledge creation model with the 
following: “the key to leading the knowledge creation process is dialectical thinking. The 
role of top management in articulating the organization’s knowledge vision is 
emphasized, as is the important role of middle management (‘knowledge producers’) in 
energizing ba.” [Nonaka 2000]. Recognizing that knowledge management is essential 
and that knowledge creation occurs and requires management begins at the top levels of 
management. Without the blessing from the top, the required focus and required assets 
will not be available for the “knowledge producers” to positively influence the process. 
Once top management sets a knowledge creation and knowledge management vision for 
an organization, the responsibility for the execution of that vision falls to middle and low- 
level management as well as to the individuals that work in that organization. Middle 
and low-level management assumes the responsibility to first, make an effort to recognize 
and identify opportunities to ‘fertilize” the knowledge sharing environment between 
personnel and processes within their purview, second, actively encourage the interaction 
of knowledge aspects, and third, emphasize the importance of knowledge creation and 
transfer to all personnel to include positive reinforcement for employee contributions in 
this area. The last point brings forward the necessity to educate, motivate, and outfit 
individuals within the organization in order to foster a knowledge creating environment, 
because new knowledge creation has its genesis at the individual level. While knowledge 
creation requires sponsorship from the top, it thrives through the “buying in” from the 
bottom coupled with active nurturing in the middle. 

Focusing on the management of the knowledge creation process leads to 
examining the knowledge creation environment closely. When the knowledge spiral was 
the model basis for knowledge creation understanding, the knowledge creation 
environment was understood to important but was explained almost superficially. A 
measured perspective on the managerial environment that exists where knowledge 
creation occurs for an organization projects multiple contradicting forces upon the 
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process. Expanding upon Figure 2, Inkpen and Dinur’s knowledge transfer classification 
framework, allows for examination of different knowledge management processes. Their 
underlying assumption is that because different knowledge transfers occur at different 
levels of the organization, knowledge management processes applied relative to the target 
transfer may provide better results than a less discriminating approach of matching 
management process to knowledge transfer or conversion. Based upon a collection of 
case studies of American firms engaged in joint ventures, Inkpen and Dinur “examined 
the processes used by firms to gain access to and transfer different types of alliance-based 
knowledge” [Inkpen 1998]. Figure 13 captures the essence of their results in correlating 
knowledge types and organizational levels [Inkpen 1998]. The first step in extracting the 
pertinent information from illustration is defining the terms and symbols used by Inkpen 
and Dinur in Figure 13 that are not universal enough to dispel any ambiguity. Conscious 
knowledge is primarily tacit in nature and individually held. Collective knowledge is 
primarily tacit; however, it is tacit knowledge that has become part of the organization’s 
tacit knowledge base. Objectified knowledge is primarily explicit and resides at the 
organizational level. JV-Parent Interaction refers to the relationship permitting 
knowledge transfers between the United States-based firms and the Japanese companies 
united through a joint business venture. Strategic Integration is the exchange of 
organizational culture and explicit knowledge sources between the joint venture partners 
in the case studies. Technology sharing knowledge is almost exclusively explicit in 
nature because it refers to information such as product design or specific process data in 
the Inkpen and Dinur study. 
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Figure 13. Knowledge Transfer Classification Network (From Inkpen 1998) 

The study referenced in Figure 13 provides a “real world” example of knowledge 
transfers that allow for expanding the theoretical knowledge base through a measure of 
reflection. A key point to consider is that the Inkpen and Dinur study focused on specific 
knowledge transfers within the confines of specific business ventures; therefore, their 
results may or may not follow the theoretical characteristics of organizational knowledge 
creation as put forth by the Nonaka research. Examining Figure 13 provides similarities 
and differences between the Inkpen and Dinur results compare against the Nonaka-based 
organizational knowledge creation theory. 

A starting point for comparing the results to theory is the overlap in the boxes 
representing knowledge transfers on Figure 13, which roughly translates to the interaction 
between knowledge-conversion modes in Nonaka’s knowledge spiral. The theoretical 
expectation is that these overlaps are where the bulk of the knowledge creation occurs as 
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knowledge spirals up from the personal level to the organizational level. A foundational 
supposition for knowledge creation is that individual knowledge is a primary seed for 
organizational knowledge creation. However, “based upon their findings, Inkpen and 
Dinur conclude that only a limited amount of knowledge associated with personnel 
transfers “spiraled” beyond the group level to the organizational level” [Inkpen 1998]. 

Perhaps an explanation for the lack of personnel knowledge spiraling up to the 
organizational level in the Inkpen and Dinur study can be attributed to the differences in 
the organizational cultures of the Japanese and American firms associated in the joint 
venture environment, which indicates the impact of ba upon the knowledge creation 
process. Inkpen and Dinur speculate that a more robust transfer of tacit knowledge at the 
organizational level was possible but not achieved. “Resistance in the American parents 
to the cost of learning limited the effectiveness of the process at the organizational level” 
[Inkpen 1998]. While this lack of knowledge spiral generated from the tacit knowledge 
held by individuals in an organization seems to contradict the Nonaka-based theory of 
organizational knowledge creation, noting the date of the Inkpen and Dinur study helps 
establish a better theoretical context for analysis of this point. The Inkpen and Dinur 
study does present findings that deviate from Nonaka based theory. However, they do 
discuss the effects of having two different cultures involved in the case studies and hint at 
the effect of that relationship on the organizational knowledge transfer and creation that 
resulted from those interactions. In effect, Inkpen and Dinur’s findings become much 
more consistent with Nonaka-based theory when analysis is expanded to include the 
importance of shared context, or ba, which was not published theory by Nonaka, 
Toyama, and Konno until 2000. Considering a breakdown or divide in the shared context 
perceived by the Japanese and American firms in the case studies presents some 
reasonable explanation as to why the “knowledge spiral” did not progress more in 
accordance with the organizational knowledge creation theories based upon the works of 
Nonaka and his associates. 

At the heart of the knowledge creation process, the SECI process is the interaction 
between the two defined poles of knowledge, i.e., tacit and explicit. This distinction 
further applies in the intended environment of this research: projecting knowledge 
creation onto an organization that traditionally relies upon regimentation and “machine- 
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like” processes. One way this interaction occurs is with the application of graduate 
education via DL while working in an operational billet, which routinely introduces 
unforeseen obstacles for the person and organization attempting to introduce new 
knowledge in explicit form into the knowledge creation process for the organization. 
Beyond the natural friction of these interactions caused by the contradictory nature of the 
forces involved is the almost tangible resistance to process and system inputs that do not 
demonstrate an intuitive increase in measurable production. For a mission-oriented 
organization, like a branch of the United States Armed Forces, these real concerns and 
interactions provide enough friction to possibly lead to a less than fully committed 
pursuit of the knowledge creation process by multiple levels of the organization. A 
breakdown in commitment anywhere in the management of the process cripples the 
potential of the process as the SECI model clearly demonstrates; therefore, understanding 
and managing the inherent contradictions in the knowledge creation environment is the 
key to success. Figure 14 depicts the knowledge spiral’s interaction with opposite or 
contradicting forces [Nonaka 2000]. 
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Figure 14. Knowledge Spiral with Contradicting Forces (Nonaka 2000) 


4. Management and Leadership 

Nonaka, Toyama, and Konno argue that the cornerstone for effective management 
of the knowledge creation process is dialectical thinking, which captures the potential 
synergy of the inherent contradiction and transcends them [Nonaka 2000]. Upon further 
consideration, the opposing or contradictory element of managing the knowledge creation 
process was stated without complete realization in the original Nonaka and Takeuchi 
model. This element of the process has always been there because it was the interaction 
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between tacit and explicit knowledge that was identified as the key element of knowledge 
creation in the initial model. Extrapolating this interaction to include all contradictory 
influence in the knowledge creation process augments the arguments put forth by 
Nonaka, Toyama, and Konno as to the importance of the dialectical element of 
knowledge management and leading for an organization. 

Discussion of managing the knowledge creation process for an organization 
highlight the total integration required from all levels of management necessary to 
succeed. Beyond management, the individuals, who are the primary means of the raw 
knowledge inputs into the knowledge creating process; have to be fully invested in the 
process as well. Nonaka, Toyama, and Konno specifically highlight the role of middle 
management as being vitally important to the success of any knowledge creation effort by 
an organization. “Especially crucial to this process is the role of the knowledge 
producers, that is, the middle managers who are at the intersection of the vertical and 
horizontal flows of information in the company and actively interact with the others to 
create knowledge by participating in and leading ba ” [Nonaka 2000]. The pivotal role of 
the middle manager makes sense because the knowledge creation process is by no means 
linear, so the best position to influence the process is the one of greatest intersection of 
system feedback loops. Because middle management generally functions at those 
intersections, it is in the best position to lead the complex process of organizational 
knowledge creation. Figure 15 is Nonaka, Toyama, and Konno’s representation of the 
leadership relationship with the knowledge creation process and depicts the many 
feedback loops plus inputs and outputs involved [Nonaka 2000]. 
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Figure 15. Leadership’s Relationship with Knowledge Creation (From Nonaka 2000) 

While Figure 15 depicts leadership’s role in the organizational knowledge 
creation process, it does not make a relative linkage between management’s role in 
organizational knowledge creation and active knowledge management as a pursuit of the 
organization. In fact, knowledge management as a managerial discipline is a separate but 
related function. For this research, knowledge management is defined as a managerial 
perspective that seeks to identify knowledge flows within an organization and then 
proactively manage those knowledge flows for the most organizational benefit. Figure 
15 is a knowledge management model that captures the relationship between knowledge 
flows and the enablers, i.e., “Culture, Infrastructure, and Technology - each active at all 
stages of knowledge flow” [Armbrecht 2001]. 

Interestingly, while the knowledge management model and the knowledge 
creation models do not model the same process, there are many parallels between them. 
The idea of knowledge sources, both tacit and explicit, providing the impetus for 
knowledge flows through an organization appears in both model types. The knowledge 
management “enablers” provide the shared context detailed in Nonaka-based models as 
ba. The Strategy and Goals block in Figure 16 correlates to the Knowledge Vision block 
presented in Figure 15. Reading the amplifying information presented in support of both 
model types does raise and emphasizes the differences in scope and focus of these 
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models. However, understanding the Nonaka-based knowledge creation models provides 
a foundation for understanding the knowledge management models. The parallels 
described lead to the assertion that effective organization knowledge management relies 
upon an understanding of the knowledge creation process for an organization 
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Figure 16. Knowledge Flows and Enablers (From Armbrecht 2001) 

A literature review of knowledge creation theory was central to the primary and 
supporting research questions of this research. It provided the base of knowledge 
necessary to conduct the data collection and data analysis portions of this research. 
Additionally, refining the scope of this literature review led to the development of the 
hypothesis curve. 


32 






III. RESEARCH QUESTIONS AND HYPOTHESIS 


A. PRIMARY RESEARCH QUESTION 

• Is NPS DL coursework comparable to resident coursework in terms of usable 
knowledge gained? 

B. SUPPORT RESEARCH QUESTIONS 

• Does NPS effectively and quickly distribute graduate - level skills via DL to the 
personnel serving in the operational billets? 

• Is there any transfer of knowledge from a DL student to their organization? 

• Can frequency of use of skills learned through DL be used as a measure of 
effectiveness for the DL program at NPS? 

C. RESEARCH METHODOLOGY 

Data gathering was accomplished through literature review, survey of DL 
students, and interviews. Review of literature comprised a significant portion of the 
research due to the large amount of data available relating to the research questions. The 
NPS DL program was the primary focus of this research because it makes graduate level 
education available to all branches of service via a DL program, which makes this 
institution a uniquely centralized platform for study of the primary research question. 
Data gathered through survey of NPS DL students was evaluated to draw conclusions 
about the effectiveness of the NPS DL program in providing usable, relevant knowledge 
to personnel serving in the operating forces. 

D. FRAMEWORK FOR BUILDING HYPOTHESIS 

Because the primary data collection tool is a survey given to active duty military 
personnel serving in the operating forces that have completed DL courses, responses 
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from that survey present an “end product” of knowledge gained through DL coursework 
from NPS. However, addressing the research questions demands a “prior state” or 
condition to compare with the “end product” to draw results and conclusions. Due to the 
unique nature of service in operating billets, i.e., rapid rotation of personnel through 
billets and commands coupled with a time constraint of approximately one year for this 
specific research effort, a “prior state” cannot be accurately measured in the response 
pool for the survey. Because a “prior state” cannot be accurately measured, a hypothesis 
of expected results from the data collected from the survey is necessary; this was 
accomplished via literature review. 

A suitable hypothesis for this research is an expected “knowledge gained” curve. 
A literature review that builds an expected knowledge gained curve must address a 
starting point at time zero, depict a rate of knowledge acquisition, a peak of usable 
knowledge gained, and an acknowledgement of knowledge decay. While an expected 
“knowledge gained” curve serves well as a hypothesis, it does not address all of the 
specific considerations required for the research questions. 

The literature review must also address the possibly unique effects of the 
personnel pool being surveyed and their environment during the learning process. The 
DL students being surveyed all are military members serving in the operational forces. 
These students are taking DL courses while serving in demanding billets with 
considerable responsibility. Additionally, the expectation is that the student pool is 
remarkably homogenous in education, motivation, and experience as compared to the 
general public due to the nature of their military officer demographic. 

1. Literature Review for Hypothesis Curve 

Prior to determining the amount of usable knowledge gained from a course or 
courses via any format or platform, a working definition of knowledge is required. Since 
this research aims to draw conclusions about knowledge gained through DL coursework 
by individuals serving in operational billets, this definition must include past and present 
experiences of the gaining individuals. Housel and Bell offer a useful definition of 
knowledge as a starting point for their work in Measuring and Managing Knowledge. 
“Knowledge is an ideational (i.e., conceptual rather than physical) construct generated 
through the agency of the human mind.” [Housel 2001]. While knowledge is generated 
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in the human mind, an important consideration is that it is created via information flow 
that is filtered and built on the individual’s belief system [Nonaka 1994]. Beyond the 
broad, working definition of knowledge presented, knowledge can be categorized further 
to better represent the human encapsulation of knowledge. Knowledge is described as 
either tacit or explicit by convention. From an education research perspective, “tacit 
knowledge is subjective, context-specific and not readily communicated other than by 
demonstration. Explicit knowledge is objective, generally applicable and capable of 
being described in systematic, propositional language.” [Hegarty 2000]. In general, for 
this research effort, explicit knowledge is gained through a more formal, classroom type 
environment, while tacit knowledge is gained through a more “hands-on” approach. 
Assuming that both explicit and tactical knowledge reinforce each other resulting in an 
effect of “the whole being greater than the sum of its parts” and that knowledge creation 
is a human endeavor influence by environment, the working definition of knowledge for 
this research is defined as the gathering and internalizing of relevant information through 
formal and informal means by an individual. Usable knowledge is defined for this 
research as knowledge applied directly or indirectly by an individual in the individual’s 
primary area of responsibility. 

a. Estimate of Knowledge at Time Zero on Hypothesis Curve 

With a working definition established for knowledge and usable 
knowledge in context of this research, the starting point for building a hypothesis 
consisting of an expected “knowledge-gained curve” is an estimation of knowledge at 
time zero. Intuitively, an individual’s knowledge at time zero is greater than zero. 
Clearly, an officer serving in an operational billet has some knowledge of his or her area 
of responsibility. A servicemember begins building a repository of tacit knowledge upon 
assumption of a billet just by virtue of performing that billet. Additionally, at a 
minimum, that servicemember likely has access to codified knowledge regarding policies 
and procedures relating to his or her current billet. Making a finite judgment on the 
amount of tacit knowledge held by one individual is not possible with any level of 
certainty due to the nature of that classification of knowledge. While specifying a level 
of tacit knowledge is not practical, tacit knowledge can be characterized. “Research has 
shown that tacit knowledge generally (a) increases with experience on the job, (b) is 


35 



unrelated to IQ, (c) predicts job performance better than IQ, (d) provides a significant 
increment in prediction above that provided by traditional tests of intelligence, 
personality, and cognitive style, and (e) overlaps across fields, though only partially” 
[Sternberg 1995]. From this characterization, an expectation exists that the tacit 
knowledge held by an individual prior to taking a DL course will vary with the billet 
held, time in that billet, and previous experience. Assuming that the knowledge gained 
from a DL course will primarily be explicit knowledge that will rest on top of the 
individual’s existing base of tacit knowledge leads to an expectation that the individual’s 
expected “knowledge-gained” curve will start at the current level of tacit knowledge and 
increase as explicit knowledge is learned. 

b. Positive Slope Characteristics of the Hypothesis Curve 
Establishing knowledge at time zero on an expected “knowledge-gained” 
curve as the sum of the tacit knowledge held by an individual, resident in the billet’s 
processes, and the explicit knowledge available to the individual in relation to the billet 
held develops a starting point for usable knowledge that is logically more than zero, less 
than some maximal value attainable, and varies from individual to individual. Adding 
knowledge through DL courses increases the total knowledge available to the individual 
at time zero and creates a situation where the expectation is an increase in knowledge of 
the individual beyond the knowledge level at time zero, which would be represented by a 
positive slope on the expected “knowledge-gained” curve from the starting point at time 
zero to some maximum knowledge value attained at some time in the future. Because the 
knowledge exposure through DL coursework is primarily explicit in nature, the increase 
in knowledge of the individual really is a question of how explicit knowledge interacts 
with the tacit knowledge held by the individual. 

2. Application of Explicit and Tacit Knowledge Interaction 

Examining how tacit and explicit knowledge interact begins with Nonaka and 
Takeuchi’s work in The Knowledge Creating Company. Their “knowledge spiral” model 
“identifies four inter-related processes by which knowledge flows around an organization 
and transmutes into different forms” as cited by Thomas Clarke and Christine Rollo 
[Clarke 2001]. While the Nonaka and Takeuchi model provides a means of 
understanding how knowledge is created, it is a model based upon industry, processes, 
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and organizations. This research, on the other hand, is primarily interested in knowledge 
creation and management at the individual level and in an environment that does not have 
the same competitive pressures or concerns as commercial industry. However, the base 
assertion of the model that tacit and explicit knowledge interact in a variety of ways to 
create new knowledge becomes the cornerstone of this research’s effort to quantify how 
much usable knowledge is gained by servicemembers in the operating forces through DL 
courses. 

The differences in environment and scope for this research and the intended 
audience for the Nonaka and Takeuchi model helps establish a bridge between the 
industrial and educational application of their model as a necessity for using their model 
to describe knowledge creation in context of this research. The key to establishing that 
bridge is defining how new knowledge is captured in a professional environment. David 
Hargreaves writes, “After knowledge has been created, it needs to be validated. In 
professional life, knowledge achieves validation when it is turned into practices which 
demonstrably and repeatedly work” [Hargreaves 1999]. If the process of refining 
practices in search of a collection of best practices in a business environment is 
substituted for the organizational processes inherent to the Nonaka and Takeuchi model, 
that model begins to have more usefulness for this research in terms of modeling and 
describing the creation and transfer of usable knowledge via the DL program. 
Additionally, Hargreaves assertion that new knowledge is validated in practice lends 
itself to using frequency of use as a metric for measuring usable knowledge transfer. 
a. Apex Characteristics of the Hypothesis Curve 
Building an argument for applying the Nonaka and Takeuchi model of 
knowledge creation as an applicable model, acknowledging concessions for this research 
in terms of knowledge creation limited to the individual and environment of those 
individuals provides the background necessary for describing the apex of the expected 
“knowledge-gained” curve. Nonaka acknowledges the transfer between explicit and tacit 
knowledge in an individual with the following: “In order to raise the total quality of an 
individual’s knowledge, the enhancement of tacit knowledge has to be subjected to a 
continual interplay with the evolution of relevant aspects of explicit knowledge” [Nonaka 
1994]. The expectation that explicit knowledge garnered through DL work will add to 
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the individuals’ existing tacit knowledge to create a positive increase in individual 
knowledge is also described in part by the Internalization component of the Nonaka and 
Takeuchi model, which models explicit knowledge conversion into tacit knowledge. 
Individuals can internalize explicit knowledge to further their reservoir of tacit 
knowledge [Nonaka 2000]. Borghoff furthers this assertion by claiming that actual new 
tacit knowledge can be created through internalizing explicit knowledge [Borghoff 1997]. 
Describing the apex of the expected “knowledge-gained’ curve centers on the interaction 
between an individual’s tacit and explicit knowledge. Since the realization of new tacit 
knowledge at the individual level does not occur until explicit knowledge has been 
internalized, the introduction of new explicit knowledge will only increase an 
individual’s level of tacit knowledge if it is internalized. Because tacit knowledge is the 
knowledge of doing and experience, the internalization process occurs primarily in the 
practical application of the new explicit knowledge. It follows that the maximum 
practical use of new skills corresponds to the maximum internalization or conversion of 
newly acquires explicit knowledge into the individual’s base of tacit knowledge. This 
assertion leads to the expectation that the apex of the expected “knowledge-gained” curve 
will correlate with the maximum frequency of use of the new explicit skills, resulting in 
the maximum amount of new total knowledge for the individual. 

b. Negative Slope Characteristics of the Hypothesis Curve 
Accepting the description that the apex of the expected “knowledge- 
gained” curve occurs at the point of maximum new explicit knowledge “added” to the 
individual’s existing base of tacit knowledge, a negative slope along the “knowledge- 
gained” curve beyond the apex is expected. This research asserts that this negative slope, 
which represents a reduction in individual knowledge, results from knowledge decay and 
from conversion of explicit knowledge gained from DL work into the tacit knowledge 
base of the individual. In general, knowledge decay results from non use of knowledge 
sets over time. Kipps, Kohen, and Paden, citing a study of knowledge decay in a group 
of economic students over various time periods, find an inverse and nonlinear 
relationship between the levels of knowledge retained at a time zero and at some time in 
the future [Kipps 1984]. The existence of knowledge decay over time is relatively easy 
to accept as a real phenomenon by any student who has ever taken a course that does not 
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directly contribute to skills necessary in the day-to-day execution of their job. The rate of 
knowledge decay is a matter that is influenced by a number of factors, such as the initial 
scholastic ability of the student and dedicated efforts over time to reinforce knowledge 
gained [Kipps 1984]. An inference from this study applies to this research because the 
assumption is that DL students will take courses that enhance specific skill sets applied in 
the routine execution of their billets. This inference leads to an expectation of a reduced 
rate of knowledge decay in this population, which should be visible in the frequency of 
use of those skills reported. Describing the negative slope of the “knowledge-gained” 
curve also includes the factor of conversion of new explicit knowledge to tacit 
knowledge. Taken from Nonaka and Takeuchi’s model, internalization, which is 
conversion of explicit to tacit knowledge, is the mode of knowledge creation that captures 
this conversion. For an individual, the interaction between tacit and explicit knowledge 
is an enabler to enhancing the individual level of experience and perspective [Nonaka 
1994]. This interaction is in reality just increasing the individual’s level of tacit 
knowledge in an area of expertise, because experience and perspective are central 
elements of an individual’s tacit knowledge base. For an expected “knowledge-gained” 
curve, this conversion of explicit knowledge to tacit knowledge will probably appear as a 
decrease in knowledge from the apex of the curve. As the usable knowledge gained is 
implemented repeatedly over time, that knowledge becomes more “intuitive” for the 
individual and will probably be used less consciously as that new knowledge is 
internalized. Because frequency of use is the metric used to draw the usable “knowledge- 
gained” curve from the survey results, this internalization aspect of the knowledge gained 
will drive the hypothesis curve to a negative slope beyond the apex of the curve. The 
final component of an expected “knowledge-gained” curve is the end point. Since the 
start of the curve at time zero is the individual’s tacit knowledge base and explicit 
knowledge is added, the internalization of that added explicit knowledge is expected to 
result in a higher level of individual tacit knowledge. 

3. Hypothesis Curve Characteristics 

From the previous literature review, the points used to construct a generalized 
hypothesis curve are as follows: 
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• knowledge level at time zero is the individual’s existing tacit knowledge 
base (e.g., four years of college + service schools + service experience, 
etc.) 

• positive slope of curve represents the addition of relevant explicit 
knowledge 

• apex of curve is defined as the maximum input of explicit knowledge prior 
to decay or internalization 

• negative slope of curve represents decay and internalization 

• end point of curve is higher than start point at time zero to reflect the 
usable knowledge gained 

• Time (T) is in months - 36 months represents a typical tour length 
The generalized expected “knowledge-gained” curve is in Figure 17. 


Tacit Knowledge Base (Hypothesis) 



Time (T) in Months 


Figure 17. Tacit Knowledge Base (Hypothesis Curve) 
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IV. DATA COLLECTION 


This chapter details the collection method used to gather data for analysis 
regarding the four research questions addressed in this study. The data collection 
methodology, population, objectives, and design are explained. 

A. DATA COLLECTION POPULATION 

1. Distributed Learning Students 

This research is concerned with the level of usable knowledge gained or refined 
through NPS DL coursework by individuals serving in the operating forces. Due to the 
student demographic required for this data collection, the first step in the process was 
identifying the student population for study. This research focuses on students that are 
currently enrolled or have completed all or part of the four classes in the Information 
Systems and Operations (ISO) Certificate Program offered through NPS DL. This 
certificate program was chosen because it is a mature program, which has over two 
hundred students that are current or past students. Additionally, this program focuses 
study in the Information Sciences discipline, which is of personal interest to the 
researcher. 

2. ISO Certificate Program 

The ISO Certificate Program is a stand alone certificate program but is also serves 
as the first phase of the Naval Postgraduate School Master of Science degree in 
Information Systems and Operations [NPS 2006]. The ISO Certificate Program consists 
of the following four courses: 

• SS3011 - Space Technology and Applications 

• 103100-Information Operations 

• IS3502 - Computer Networks: Wide Area/Local Area (Intro to Information 
Systems Networks) 

• CC3000 - Intro to Command , Control, Communication, Computer and 
Intelligence Systems in DoD 

Individual course descriptions provide a higher level of insight into the scope of 
the ISO Certificate Program. Individual course descriptions accessible through the ISO 
Certificate Program webpage follow: 
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SS3011 - Space Technology and Applications 

An introduction to space mission analysis with an emphasis on those space 
missions supporting military operations. Topics include space history, 
doctrine and organizations, orbital mechanics, communication line 
analysis, space environment, spacecraft technology, and military, civil and 
commercial space systems [NPS 2006]. 

103100 - Information Operations 

This course provides a survey of Information Operations (IO) along the 
time line of peace, to conflict, and back to cessation of hostilities. Students 
study the methods and elements which contribute to successful 
Information Operations including: Psychological operations and 
deception, Operational security, information assurance, and infrastructure 
protection, Electronic attack/protect/support, Physical attack/destruction in 
support of IO, Military-civilian relationship, Human cognition and 
decision making. Command and control structures, Legal issues, 
Computer and network attack, Systems engineering concepts (including 
modeling and simulation), Sensor and signals intelligence support to IO 
[NPS 2006], 

IS3502 - Computer Networks: Wide Area/Local Area 

Architecture, standard protocols, and technological advances in computer 
networks, with an emphasis on internet working and interoperability. 
Specific topics include open network architectures (OSI vs. DoD 
architecture), X.25, local area networks, TCP/IP, and a variety of 
distributed application services built on the client-server model. Students 
also gain an understanding of Network Centric Warfare requirements 
surrounding DDN (Defense Data Network), X.400-based DMS (Defense 
Message System), SDNS (Secure Data Network Service), and GOSIP 
(Government Open System Interconnection Profile) [NPS 2006]. 

CC3000 - Intro to Command, Control, Communication, Computer and 
Intelligence 

Systems in DoD Knowledge of current C4I systems and practice is introduced. A 
basic framework for understanding C4I is provided. Case studies are used as well 
as lessons learned from crises, field exercises and wargaming [NPS 2006]. 
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B. DATA COLLECTION DESIGN 

Because the primary data collection tool for this research is a survey, the design 
of that survey required planning to ensure reliable data collection. The primary concern 
was designing questions that elicited responses directly related to the four research 
questions in this study. After drafting questions for these areas of interest, the survey 
required six iterations of review in order to scrub leading or ambiguous questions. The 
desire was to eliminate as much of the potential for respondent bias or confusion as 
possible. 

1. Initial Survey Design 

The survey design process began with a previous set of questions from a survey 
developed for the ISO program students. This survey was primarily generated to capture 
a mix of student demographic information along with student perceptions regarding the 
training/education mix and delivery of graduate education via DL in the ISO program. 
The version of the initial questions employed in the final survey is presented in Table 1. 
The choices of answers were checkboxes, radio buttons, and rating tables, respectively. 


No. 

Question 

1 

Check each course below taken through DL with NPS. 

2 

How long has it been since you took your last DL course(s)? 

3 

In what context were you taking DL coursework? 

4 

Estimate the split between amount of training vice education in 

the course(s) you have taken. For example, if a course was a quarter 

training and three quarters education (theory), select 25%-75%. Select 

N/A if course was not taken. 

5 

During the course(s), which learning context did you like the 

most? 

6 

How much of the course(s) material applies to your current job? 

Select N/A if course was not taken. 

7 

Rate your confidence in subject matter since taking the course. 

Select N/A if course was not taken. 

8 

Rate the quality of the DL course(s) in terms of providing 
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applicable skills for your billet. Select N/A if course was not taken. 

9 

Rate your level of Subject Matter Expertise before taking DL 

course(s). Select N/A if course was not taken. 

10 

Rate your use of knowledge gained or refined from DL course(s) 

at work. Select N/A if course was not taken. 


Table 1. Initial Survey Questions 

With the existing questions presented in Table 1 as a base, the survey design 
effort turned to developing a survey that captured all of the areas of interest of this 
research. Specifically, the survey needed to query for billet application of DL education, 
usable knowledge gained or refined, tacit knowledge base, and transfer of knowledge to 
the organization in order to provide data to address the four research questions. The 
survey generation process began in October and consisted initially of interaction between 
the researcher and the Co-Thesis Advisor, Steve Iatrou. Because Steve Iatrou is the 
Academic Advisor for ISO certificate students, he has special insight into the students 
and program that helped in the refinement of the survey. The design process went 
through six iterations from the beginning of October until the beginning of December. 
After the six refinements were completed, the survey reached its final length of twenty- 
one questions. 

With a refined survey design, the research concern became researcher bias in the 
questions. Because the architects of the survey were the researcher and the Academic 
Advisor of the program, the researcher conducted a limited usability test with a subject to 
determine the basic readability of the test. The test subject has a Masters Degree in 
Psychology and works as a counselor at California State University at Monterey Bay. 
She had no working knowledge of the knowledge creation process or knowledge transfer 
theory. She also has never taken a DL Course. The researcher presented the survey to 
the subject and asked her to detail anything that was not clear to her in the instructions. 
After reviewing the survey, her main concern was the definition of tacit knowledge. The 
initial definition provided was, “tacit knowledge is subjective, context-specific and not 
readily communicated other than by demonstration.” [Hegarty 2000]. After her input, the 
tacit knowledge definition provided for the survey was changed to reflect confidence in a 
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subject matter as well. The ultimate definition is as follows: “Tacit knowledge is defined 
for this study as the level of functional knowledge that you posses that is comprised of 
the summation of your experience, both personal and professional, your level of expertise 
gained through working in your billet over a period of time, and your lifelong collection 
of formal and informal education and training experiences. Tacit knowledge is 
manifested in your degree of confidence in performing specific tasks or billets, i.e., your 
level of confidence in being “the ‘subject matter expert’ for a specific role in your 
organization.” 

Once the survey was updated to reflect the “outsider” input, the survey was tested 
against a Professor in the IS Department at NPS. The Professor chosen for this review 
has extensive experience in questing for qualitative data but had no previous direct 
interaction with this research effort. These reviews were primarily conducted to get a 
professional review of the survey question design, i.e., are the questions clear and are 
they asking the right question? This Professor’s input consisted of minor verbiage 
changes and guidance on overall survey construction. After this review, the survey was 
modified in terms of layout to group like questions. Also, each research question was 
addressed by two groups of questions that queried to the same question but in slightly 
different ways to check for intra-survey reliability, consistency in responses. The intent 
of these layout modifications was to identify disconnects between groups of questions 
that went to the same research question. That is, if two sets of targeted questions have a 
high correlation, then the data is viable. If two sets of targeted questions have a low 
correlation, then the data is unreliable for drawing conclusions and inferences. 

Completing these mini-reviews led to the development of a survey that was ready 
for pre-testing. The survey questions are in Table 2. The choices of answers were 
checkboxes, radio buttons, and rating tables, respectively. 


No. 

Question 

1 

Check each course below taken through DL with NPS. 

2 

How long has it been since you took your last DL course(s)? 

3 

1 n what context were you taking DL coursework? 

4 

Estimate the split between the amount of training vice education 
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in the course(s) you have taken. For example, if a course was a quarter 

training and three quarters education (theory), select 25%-75%. Select 

N/A if course was not taken. 

5 

During the course(s), which learning context did you like the 

most? 

6 

How much of the course(s) material applies to your current job? 

Select N/A if course was not taken. 

7 

Rate your confidence in subject matter since taking the course. 

Select N/A if course was not taken. 

8 

Rate the quality of the DL course(s) in terms of providing 

applicable skills for your billet. Select N/A if course was not taken. 

9 

Rate your level of Subject Matter Expertise before taking DL 

course(s). Select N/A if course was not taken. 

10 

Rate your use of knowledge gained or refined from DL course(s) 

at work. Select N/A if course was not taken. 

11 

Prior to taking your DL coursework, how often were you called 

upon to use skills later refined through a DL course in the fulfillment of 

your operational billet duties for the following courses? Select N/A if 

course was not taken. 

12 

How often did you use those skills gained or refined WHILE 

taking a DL course in fulfillment of your operational billet responsibilities 

for the following courses? Select N/A if course was not taken. 

13 

AFTER completing a DL course, how often did you use those 

specific skills gained or refined in the fulfillment of your operational billet 

duties for the following courses? Select N/A if course was not taken. 

14 

What level of usable knowledge gained or refined from your DL 

coursework was completely new to you for the following courses? Select 

N/A if course was not taken. 

15 

What level of usable knowledge gained from your DL coursework 

was a refinement of existing knowledge for you from the following 

courses? Select N/A if course was not taken. 

16 

In terms of your tacit knowledge base, rate your mastery level of 

the skill sets learned through DL work for the following scenarios 

regarding 10 3100? Do not answer this question if you have not taken 

10 3100. Select N/A for scenarios that do not apply; e.g. if you are 
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currently taking 10 3100, select N/A for the scenarios based AFTER 

taking a course and one year AFTER taking a course. 

17 

In terms of your tacit knowledge base, rate your mastery level of 

the skill sets learned through DL work for the following scenarios 

regarding SS 3011? Do not answer this question if you have not taken 

SS 3011. Select N/A for scenarios that do not apply; e.g. if you are 

currently taking SS 3011, select N/A for the scenarios based AFTER 

taking a course and one year AFTER taking a course. 

18 

In terms of your tacit knowledge base, rate your mastery level of 

the skill sets learned through DL work for the following scenarios 

regarding 1S 3502? Do not answer this question if you have not taken 1S 

3502. Select N/A for scenarios that do not apply; e.g. if you are 

currently taking SS 3011, select N/A for the scenarios based AFTER 

taking a course and one year AFTER taking a course. 

19 

In terms of your tacit knowledge base, rate your mastery level of 

the skill sets learned through DL work for the following scenarios 

regarding CC 3000? Do not answer this question if you have not taken 

CC 3000. Select N/A for scenarios that do not apply; e.g. if you are 

currently taking SS 3011, select N/A for the scenarios based AFTER 

taking a course and one year AFTER taking a course. 

20 

How much of new knowledge acquired through DL coursework 

was incorporated into procedures, directions, SOPs, OJT, informal 

training, etc. within your work section WHILE you were taking one of the 

foilwing DL courses? 

21 

How much of new knowledge acquired through DL coursework 

was incorporated into procedures, directions, SOPs, OJT, informal 

training, etc. within your work section AFTER you took one of the 

following DL courses? 


Table 2. Pre-Test Survey Questions 

2. Survey Pre-Test and Results 

After refining the survey as described, a pre-test was conducted prior to fielding 
the survey. The pre-test was applied to eight resident students at NPS from four different 
curricula. Each resident student has taken at least one of the resident equivalent courses 
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corresponding to the respective courses in the ISO curriculum. The resident curricula 
represented in the pre-test were Information Technology Management, Space Systems, 
Information Operations, and Joint Command and Control Systems. Each curriculum was 
represented by two students. Two of the students had taken all four of the resident 
equivalent classes in the ISO program. Four of the students had taken two of the classes. 
Two of the students had taken one of the classes. The resident students were asked to 
answer the questions and to note any instruction that was not clear. The pre-test was 
fielded with a brief email introduction but no additional instruction other than the survey 
questions themselves and a definition of tacit knowledge for this survey. The email 
instructions were as follows: 

Special Instructions for Pre-Test 

Gentlemen, 

The purpose of this pre-test is to help me validate the clarity of the instructions on 
my survey and exercise the survey tool mechanisms. The DL courses are listed below 
with their resident equivalents. I am asking you for your input because you have taken 
some or all of the target courses in a resident or DL setting. The aim of my research is to 
quantify the amount of usable knowledge gained by personnel taking DL coursework 
through NPS while serving in operational billets. 

Course Equivalents 


10 3100 IW 3101 Introduction to Information Operations 
SS 3011 -> SS 3011 Introduction to Space Operations 
IS 3502 IS 3502 Introduction to WAN/LAN Networking 
CC 3000 CC 3000 Introduction to Command and Control 


-General Instructions- 

These are the basic instructions for these surveys are as follows: 

DL is acronym for Distributed Learning. 

Note: Tacit knowledge is defined for this study as the level of functional 

knowledge that you posses that is comprised of the summation of your experience, both 

personal and professional, your level of expertise gained through working in your billet 

over a period of time, and your lifelong collection of formal and informal education and 

training experiences. Tacit knowledge is manifested in your degree of confidence in 
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performing specific tasks or billets, i.e., your level of confidence in being “the subject 
matter expert” for a specific role in your organization. 

The step-by-step is as follows: 

• Click on this link http://131.120.251.62/QCL/survey.asp?survey=75 
(December 2005) 

• Questions 1-6 and #18 are demographic questions, so just click any radio 
button or checkbox. 

• Assume a “willing suspension of belief’ in regards to the rest of the 
questions and take a SWAG at an answer as if you were in the Fleet and 
taking these courses via DL. 

• If the question is about a course you have not taken, please respond with a 
N/A. 

• Click the FINISH button at the bottom of the survey. 

• Please send me an email with either specific questions or parts of 
questions that did not make sense to you or an email saying you 
understood the directions as-is. 

Of course, none of your responses will be used in any manner other than indicated 
in the above instructions. You can expect complete anonymity as to your responses; in 
fact, the survey tool does not collect your name or email information. Thanks for giving 
me a hand with this. 

R/S 

Scott Hortman 

3. Final Survey Pretest Results 

The resulting comments were synthesized for editing the survey design as 
follows: 

• Each respondent reported that the layout of the survey made reading the 
questions more difficult than it needed to be. 

• Some respondents reported trouble getting access to the survey. 

• Multiple respondents did not like the various factors involving the answer 
choices. Specifically, negative trends that were highlighted in the pretest 
regarding answers were lack of uniformity in scale between some sets of 
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answers, lack of a N/A or “all of the above” option on some answers, and 
lack of a consistent grouping of like subjects. 

• No respondent reported significant “readability” or “understandability” 
issues beyond an assertion that grouping like subjects would make it easier 
for a survey recipient to answer like subjects with consistency. 

These trends were addressed in the Final Survey Design as follows: 

• The design layout and connectivity issues were addressed by moving the 
final survey to a professionally operated and maintained survey tool. 

• The answer portions of the final survey were changed to achieve a more 
consistent, user-friendly characteristic in accordance with the comments 
from the pretest results. 

• The questions were grouped in a logical way that supported the research 
questions and demographic information. 

4. Final Survey Design 

The final survey design is presented in screenshots in exactly the way the survey 
was presented to recipients. The screenshots are in the order in which those respective 
screens appear when taking the survey. The final survey is as follows: 
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Figure 18. Final Survey Introduction 

The Introduction page introduces the research to the recipient and gives information such 
as definitions, length of survey, and special response direction designed to aid the user in 
taking the survey with minimal confusion. 
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After clicking “NEXT”, the user arrives at this page. 



NPS Distance Learning Survey Exit this 


2. Minimal Risk and Privacy Statement 

1. Introduction: You are invited to participate in a study entitled: Usable Knowledge Gained Through Distance Learning at the Naval 
Postgraduate School being conducted at the Naval Postgraduate School. 

Please read the following statements before proceeding. 

Procedures: If I agree to participate in this study, I understand I will be provided with an explanation of the purposes of the research, a 
description of the procedures to be used, identification of any experimental procedures, and the expected duration of my participation. The 
entire survey should take 10-15 minutes to complete. 

Risks and Benefits: I understand that this project does not involve greater than minimal risk and involves no known reasonably foreseeable 
risks or hazards greater than those encountered in everyday life. I have also been informed of any benefits to myself or to others that may 
reasonably be expected as a result of this research. 

Compensation: I understand that no tangible reward will be given. I understand that a copy of the research results will be available upon 
request at the conclusion of the experiment, 

Confidentiality & Privacy Act: I understand that all records of this study will be kept confidential and that my privacy will be safeguarded. No 
information will be publicly accessible which could identify me as a participant, and I will be identified only as a code number on all research 
forms. I understand that records of my participation will be maintained by NPS for five years, after which they will be destroyed. 

Voluntary Nature of the Study: I understand that my participation is strictly voluntary, and if I agree to participate, I am free to withdraw at 
any time without prejudice. I also understand I may refuse to answer any particular question. 

Points of Contact: I understand that if I have any questions or comments regarding this project upon the completion of my participation, I 
should contact the the researcher, Captain Billy S. Hortman at bshortma@nps.edu or the Principal Investigator, Professor Thomas J. Housel at 
tjhousel@nps.edu. 

Statement of Consent: I have read and understand the above information, including the nature and purpose of this study, the risks and 
benefits involved, as well as the option to withdraw at any time. I have asked all questions and have had my questions answered, 

By clicking on the "Yes" button, I am acknowledging that I have read and understand this information and that I agree to voluntarily 
participate in this online survey. I also understand that I may discontinue at any time simply by exiting this website. 


1. Choose one of the following: 

J Yes 

„) No 

<< Prev Next >> _ 

__ . _ g 

j ej Done [ [ g Internet 


Figure 19. Minimal Risk and Privacy Statement 

The “Minimal Risk and Privacy Statement” page provides the users with their rights 
involved with this research and provides a feedback mechanism for them to agree to 
acknowledge their consent prior to beginning the survey. 
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After clicking “NEXT”, the user arrives at this page 


■3 NPS Distance Learning Survey - Microsoft Internet Explorer 


File Edit View Favorites Iools Help 


O Back • J- i*l I»1 fi)| Search Favorites | 


m □ a -a 


Address |^) http://www.surveymonkey.com/Users/15384796/Surveys/395011681133/CE4F81DF-FEE9-4FE4-B3F3-FB7D26CC0144. asp?U=395011681133&DO NOT COPY THIS LINK 
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NPS Distance Learning Survey 


3. Survey Section (Demographic Information) 




2. Check each course below taken through distance learning (DL) with NPS. 

f 10 3100 - Information Operations 
T~ SS 3011 - Space Operations 
IS 3502 - Network Operations 
r CC 3000 - Command and Control 

3. How long has it been since you took your last DL course(s)? 

J 1-6 months 
J 6months - 1 year 
J 1 year - 2 years 
J 2 years - 3 years 
J 3 years or longer 

4. In what context were you taking DL coursework? 

As part of a degree program 
As part of a certificate program 
As a stand alone course 

5. In the following table, estimate the split between amount of training vice education in the course(s) you have taken. For example, if a course was a quarter 
training and three quarters education (theory), select 25 < H>-75%. Select N/A if course was not taken. 



N/A 

100%-0% 

75%-25% 

S0%-50% 

25%-7i 

IO 3100 - Information Operations 


J 

J 

J 

J 

SS 3011 - Space Operations 

J 

J 

J 

J 

J 

IS 3502 - Network Operations 

J 

J 

J 

J 

J 

CC 3000 - Command and Control 

J 

J 

J 

J 

J 


6. During the course(s), which learning context did you like the most? 

Interaction - exchange with fellow students 
J Interaction - exchange with instructor 
J Working alone 


| cj javascript: NextFunction(); 


Figure 20. Demographic Information 

The reason for collecting demographic information is to help provide insight into the 
survey responses during data analysis. 
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After clicking “NEXT”, the user arrives at this page. 


3 NPS Distance Learning Survey - Microsoft Internet Explorer 


File Edit View Favorites Tools Help 




Q> 


SHI £ 2 ) Search Favorites el 


, h □ a -a 


Address |,j*] http://www.surveymonkey.com/Users/15384796/Surveys/395011681133/BE8B3913-CE8E-43A4-8D79-6D81BF734286.asp?U=3950116811338d30 N0T C0PY THIS LINK 
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NPS Distance Learning Survey 


4. Survey Section (Billet Application) 

7. How much of the course(s) material applies to your current job? Select N/A if course was not taken. 

N/A 0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 


10-90% 90-100% 


IO 3100 

J 

J 

y> 

J 

■J 

J J J J 

yj 

J 

SS 3011 

J 

J 

J 

J 

J 

j J j J 

yj 

J 

IS 3502 

J 

J 

J 

J 

J 

j J j J 

J 

yj 

CC 3000 


J 



J 

J J J J 

yj 

■J 

8 . Rate your confidence in subject matter since taking the course. Select N/A if course was not taken. 

HIGH MEDIUM . .... 

HIGH MEDIUM MEDIUM LOW L ° W N/A 



IO 3100 

J 

J 

J 

yj 

J 

■J 



SS 3011 

J 

J 

J 

J 

J 

J 



IS 3502 

J 

J 

J 

j 

-7 

J 



CC 3000 

J 

J 

-> 

J 

-> 

J 



9. Rate the quality of the DL course(s) in terms of providing applicable skills for your billet. Select N/A if course was 

Lira'Ll HIGH uer.Tiu.il MEDIUM | _..... 

HIGH MEDIUM MEDIUM LOW L0W N/A 

not taken. 

IO 3100 

yj 

J 

yj 

J 

J 

J 



SS 3011 

J 

J 

J 

J 

J 

J 



IS 3502 

J 

-J 

J 

yJ 

J 

J 



CC 3000 


J 


J 


J 



10. Rate your level of Subject Matter Expertise before taking DL course(s). Select N/A if course was not taken. 

uiTi-LJ HIGH uenn IU MEDIUM | _... .... 

HIGH MEDIUM MEDIUM LOW L0w N/A 


IO 3100 

J 

J 

J 

J 

J 

J 



SS 3011 

J 

J 

J 

J 

J 

J 



IS 3502 

yj 

yj 

yj 

J 

yJ 

yj 



CC 3000 

J 

J 

■J 

J 

J 

J 




11 . Rate your use of knowledge gained or refined from DL course(s) at work. Select N/A if course was not taken. 
[cj Done 


~rrrrri¥ Ir 


Figure 21. B illet Application 


All of these questions query to the billet application of knowledge gained or refined 
through DL coursework. 


54 














This page is a continuation of the previous page. Because the screen requires a scrollbar 
to view it in its entirety, the screen was adjusted to capture question 11 and its associated 
responses for this screenshot. 
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8. Rate your confidence in subject matter since taking the course. Select N/A if course was not taken. 



HIGH 

HIGH 

MEDIUM 

MEDIUM 

MEDIUM 

LOW 

LOW 

N/A 

10 3100 

J 

J 

J 

J 

J 

J 

SS 3011 

J 

J 

J 

J 

J 

J 

IS 3502 

J 

J 

J 


J 

J 

CC 3000 

J 

J 

J 

J 

J 

J 

9. Rate the quality of the DL course(s) in 

terms of providing applicabli 


HIGH 

HIGH 

MEDIUM 

MEDIUM 

MEDIUM 

LOW 

LOW 

N/A 

10 3100 


J 


J 

J 

J 

SS 3011 

J 

J 

J 

J 

J 

J 

IS 3502 

J 

J 


J 

J 

J 

CC 3000 

J 

J 

J 

J 

J 

J 


10. Rate your level of Subject Matter Expertise before taking DL course(s). Select N/A if course was not taken. 



HIGH 

HIGH 

MEDIUM 

MEDIUM 

MEDIUM 

LOW 

LOW 

N/A 

IO 3100 


J 

J 




SS 3011 

J 

J 

J 

J 

J 

J 

IS 3502 

J 

J 

J 

J 

J 


CC 3000 

J 

J 

J 

J 

J 

J 

11. Rate your use 

' of knowledg 

e gained or 

refined from DL co 

iurse(s)a 


HIGH 

HIGH 

MEDIUM 

MEDIUM 

MEDIUM 

LOW 

LOW 

N/A 

IO 3100 

J 

J 

J 

J 

J 


SS 3011 


J 

J 

J 


J 

IS 3502 

J 

J 

J 

J 

J 

J 

CC 3000 

J 

J 

J 

J 

J 

J 


| | | !• Internet 


Figure 22. Billet Application Continued 
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After clicking “NEXT”, the user arrives at this page 



Figure 23. Usable Knowledge Gained or Refined 

This section is questioning how much usable knowledge was gained or refined through 
DL coursework in specific courses and in specific times relative to taking the course(s). 
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This page is a continuation of the previous page in order to capture every question in this 
section. 
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13. How often did you use those skills gained or refined WHILE taking a DL course i 
Select N/A if course was not taken. 

fulfillment of your 

-3 

perational billet responsibilities for the following courses? 


N/A 

0-10% 

10-20% 

20-30% 

30-40% 

40-50% 

50-60% 

60-70% 

70-80% 

80-90% 

90-100% 

10 3100 


J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

SS 3011 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

IS 3502 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

CC 3000 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 


14. AFTER completing a DL course, how often did you use those specific skills gained or refined in the fulfillment of your operational billet duties for the following 
courses? Select N/A if course was not taken. 


N/A 

0-10% 

10-20% 

20-30% 

30-40% 

40-50% 

50-60% 

60-70% 

70-80% 

80-90% 

90-100% 

IO 3100 

J 

J 

J 


J 

J 

J 

J 

J 

-> 


SS 3011 

J 

J 

J 

J 

J 

J 

J 


J 

J 

J 

IS 3502 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

CC 3000 

J 

J 

J 

J 

J 

J 

- J 

J 

J 

J 


15. What level of usable knowledge gained or refined from your DL coursework was completely new to you for the following courses? Select N/A if course was not 
taken. 


N/A 

0-10% 

10-20% 

20-30% 

30-40% 

40-50% 

50-60% 

60-70% 

70-80% 

80-90% 

90-100% 

IO 3100 

J 


J 

J 

J 

J 

J 

J 

J 

J 

J 

SS 3011 

J 

J 

J 

J 

J 

J 

j 

J 

J 

J 

J 

IS 3502 

J 

J 

J 

J 

J 

J 


J 

J 

J 

J 

CC 3000 

J 

J 

J 

J 

J 

J 

-* 

J 


J 

J 

16. What level of usable knowledge gained from your DL coursework was a refinement of existing knowledge for you from the following courses? Select N/A if course 
was not taken. 


N/A 

0-10% 

10-20% 

20-30% 

30-40% 

40-50% 

50-60% 

60-70% 

70-80% 

80-90% 

90-100% 

IO 3100 

J 


J 


J 

J 

J 

J 

J 


J 

SS 3011 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

J 

IS 3502 

J 

J 

J 

J 

J 

J 

J 

J 


J 

J 

CC 3000 



J 


J 

J 

J 

J 


J 

J 
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Figure 24. Usable Knowledge Gained or Refined Continued 
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After clicking “NEXT”, the user arrives at this page 
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NPS Distance Learning Survey 

Exit this survey 


1 6. Survey Section (Tacit Knowledge Base) 


17. In terms of your tacit knowledge base, rate your mastery level of the skill sets learned through DL work for the following scenarios regarding IO 3100? Do not 
answer this question if you have not taken IO 3100. Select N/A for scenarios that do not apply; e.g. if you are currently taking IO 3100, select N/A for the scenarios 
based AFTER taking a course and one year AFTER taking a course. 

0-20% 20-40% 40-60% 60-80% 80-100% N/A 

Prior to taking IO 3100 

While taking IO 3100 ,J J J J J 

After taking IO 3100 J J j j j j 

One year AFTER taking IO 3100 J 

18. In terms of your tacit knowledge base, rate your mastery level of the skill sets learned through DL work for the following scenarios regarding SS 3011? Do not 
answer this question if you have not taken SS 3011. Select N/A for scenarios that do not apply; e.g. if you are currently taking SS 3011, select N/A for the scenarios 
based AFTER taking a course and one year AFTER taking a course. 

0-20% 20-40% 40-60% 60-80% 80-100% 

Prior to taking SS 3011 j ^ J 

While taking SS 3011 J J 

After taking SS 3011 ,J J J J J 

One year AFTER taking SS 3011 

19. In terms of your tacit knowledge base, rate your mastery level of the skill sets learned through DL work for the following scenarios regarding IS 3502? Do not 
answer this question if you have not taken IS 3502. Select N/A for scenarios that do not apply; e.g. if you are currently taking SS 3011, select N/A for the scenarios 
based AFTER taking a course and one year AFTER taking a course. 

0-20% 20-40% 40-60% 60-80% 80-100% N/A 

Prior to taking IS 3502 ~J -J ~J -J 

While taking IS 3502 J J J 

After taking IS 3502 J J J j 

One year AFTER taking IS 3502 J J 

20. In terms of your tacit knowledge base, rate your mastery level of the skill sets learned through DLwork for the following scenarios regarding CC 3000? Do not 
answer this question if you have not taken CC 3000. Select N/A for scenarios that do not apply; e.g. if you are currently taking SS 3011, select N/A for the scenarios 
based AFTER taking a course and one year AFTER taking a course. 

0-20% 20-40% 40-60% 60-80% 80-100% N/A 

Prior to taking CC 3000 

While taking CC 3000 J J J J J J 


|4J Done 
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Figure 25. Tacit Knowledge Base 


This section asks the recipient questions targeted to capture the effects of DL coursework 
on their tacit knowledge base time-respective of when they took a course(s). 
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This is a continuation of the previous screen, 



Figure 26. Tacit Knowledge Base Continued 
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After clicking “NEXT”, the user arrives at this page. 



Figure 27. Transfer of Knowledge to Organization 

These questions are concerned with how much knowledge gained or refined through DL 
coursework is transferred to the organization. Clicking “NEXT” sends the recipient to a 
conclusion screen that thanks them for their participation. 

C. DATA COLLECTION OBJECTIVES 

Beyond the esthetics of the design layout, the survey design had to query for data 
that would provide insight into the research questions and minimize the very real 
limitation of self-report that is inherent to a survey as a data collection tool. Additionally, 
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the fielding procedure required a concentrated effort to reach as much of the target 
population as possible and minimize impact on survey recipient’s schedule. 

1. Map Survey to Research Questions 

The final survey was designed to group like questions that queried for data that 
would be relevant to a specific research question. The “Billet Application” section of the 
survey shown in Figures 21 and 22 are intended to map to the research question “Does 
NPS effectively and quickly distribute graduate - level skills via DL to the personnel 
serving in the operational billets?” The “Tacit Knowledge Base” section of the survey 
shown in Figures 25 and 26 are intended to map to the research question “Is NPS DL 
coursework comparable to resident coursework in terms of usable knowledge gained?” 
The “Transfer to Organization” section of the survey shown in Figure 27 is intended to 
map to the research question “Is there any transfer of knowledge from a DL student to 
their organization?” The “Usable Knowledge Gained or Refined”” section of the survey 
shown in Figures 23 and 24 are intended to map to the research question “Can frequency 
of use of skills learned through DL be used as a measure of effectiveness for the DL 
program at NPS?” 

2. Control Measures to Minimize or Identify Self Report 

Because a survey was the primary data collection means used in this research and 
because self-report is a potential weakness in a survey, the survey used was designed to 
ask for the desired data in two places in the survey with slightly different wording. One 
question set will query directly to the desired data response. Its corresponding question 
set queries to the underlying concept. The design is intended to mask the direct 
association of the question sets so that the recipient does not recognize the association. 
The desire is that after the data is collected those pairs of question sets can be correlated. 
The assumption is that a high correlation between respective data sets leads to a higher 
level of confidence in the direct query question set’s respondent data. The converse is 
expected as well. The questions sets associated with their respective research questions 
are as follows: 

1) Does NPS effectively and quickly distribute graduate - level skills via DL to the 
personnel serving in the operational billets? 
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• Primary Survey Questions: Survey Section (Billet Application) Questions 7-11 

• Associated Survey Questions: Questions 12, 13, and 14 

The primary question set for this research question does not differentiate between explicit 
or tacit knowledge. The questions focus on applicable skills gained in relation to billet, 
the degree on confidence in the subject material time relative to taking the course(s), and 
use of knowledge gained in performance of billet duties. The associated question set 
queries specifically to the frequency of use of skills gained or refined through DL 
coursework before, during, and after taking a course(s). The expectation is that responses 
about how often new knowledge is used at work (frequency of use) will correlate with 
billet specific responses from the primary question set. 

2) Is NPS DL coursework comparable to resident coursework in terms of usable 
knowledge gained? 

• Primary Survey Questions: Survey Section (Tacit Knowledge Base) 

Questions 17-20 

• Associated Survey Questions: Questions 8 and 10 

Each class is individually compared respective of this survey section to the hypothesis 
curve because the hypothesis curve demonstrates the expected growth of an individual’s 
tacit knowledge base during DL coursework. If the individual course curves correlate 
with the hypothesis curve, then the suggestion that DL is comparable to resident 
coursework in terms of usable knowledge gained because the hypothesis curve was built 
from a literature review that was not context specific. Additionally, the primary question 
set queries tacit knowledge base within the given definition for each course in relation to 
the following: prior to taking the course, while taking the course, after taking the course, 
and one year after taking the course. The associated question set asks for responses 
regarding confidence and level of subject matter expertise before and after the course(s). 
The expectation is that the associated question set will correlate with the primary set for 
before and after time periods because the terms “confidence” and “subject matter 
expertise” were used in the tacit knowledge definition included for this survey. 
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3) Is there any transfer of knowledge from a DL student to their organization? 

• Primary Survey Questions: Survey Section (Transfer of Knowledge to 
Organization) Questions 21-22 

• Associated Survey Questions: Questions 7, 12, 13, and 14 

The primary question set asks specifically how much of the new knowledge gained or 
refined through DL coursework was incorporated into procedures, SOPs, OJT, etc. within 
the student’s work section. Essentially, how much of the new knowledge gained or 
refined was transferred to explicit form by the student to remain in the work section. The 
associated question set queries frequency of use of skills and how much of the course 
material applies to the current billet. The expectation is that knowledge identified as high 
frequency of use would affect processes enough that some of that new knowledge would 
need to be codified within that individual’s work section. 

4) Can frequency of use of skills learned through DL be used as a measure of 
effectiveness for the DL program at NPS? 

• Primary Survey Questions: Survey Section (Usable Knowledge Gained or 
Refined) Questions 12-16 

• Associated Survey Questions: Questions 9 and 11 

The primary question set queries frequency of use specifically. The associated question 
set asks for billet applicable skills learned and use of knowledge gained or refined 
through DL coursework. The expectation is that the associated question set will tightly 
correlate with frequency of use because the questions are work performance related. 

3. Survey Fielding Procedure 

After completing the survey design, fielding the survey was the next step. 
Because the population for this survey is so diverse geographically and because a 
significant portion this population may be busy in the operating forces, a key concern in 
fielding the survey was to minimize impact upon the recipients. In order to maximize 
responses, the Academic Associate for the 271 curriculum sent out a preparatory email to 
the distribution list explaining the intent and importance of this survey on the first Lriday 
in Lebruary. The survey was sent to the distribution list on the following Monday. The 
survey remained open for two weeks. At the end of the two week period, an email was 
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sent to the distribution list thanking those that had responded and encouraging recipients 
to complete the survey if they had not responded. The survey was closed at the end of 
three weeks. Only two additional responses were collected during the final week. 

D. DATA COLLECTION RESULTS 


The data collection results focus on two elements, the survey response rate and 
the actual raw data results. The response rate is important in order to form a judgment on 
the validity of the results. The actual raw data is important because it becomes the basis 
for the analysis. 

1. Response Rate 

The total population was estimated at 236 students. This figure came from the 
academic records of the Academic Advisor for the 271 curriculum. The entire population 
was placed upon the survey distribution list. Out of the 236 surveys distributed, 54 came 
back as undeliverable, resulting in an actual reachable population of 182. An important 
observation is that contact information for these students is not maintained in a current 
status. For example, former students that used government email accounts during DL 
coursework and have since moved to a different job are not reachable. Out of the 182 
surveys that were received, 38 students answered the survey. This translates to a 20% 
response rate from the reachable population. Some of the questions were not answered 
by every respondent, but that information is captured in the raw data. 

2. Raw Data Results 

The raw results are presented in the following screenshots. 
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Figure 28. Raw Survey Results 
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Figure 29. Raw Survey Results Continued 
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5. In the following table, estimate the split between amount of training vice education in the course(s) you have taken. For example, if a 
course was a quarter training and three quarters education (theory), select 2596-75%. Select N/A if course was not taken. 


N/A 

100A6-0A6 

75^-25^ 

50<Vb-50At) 

25<W>-75<M> 

Response 

Total 

IO 3100 - Information Operations 

9% (3) 

6% (2) 

11% (4) 

14% (5) 

60At> (21) 

35 

SS 3011 - Space Operations 

17% (6) 

6% (2) 

6% (2) 

23% (8) 

49<K> (17) 

35 

IS 3502 - Network Operations 

23% (8) 

9% (3) 

14% (5) 

26% (9) 

29% (10) 

35 

CC 3000 - Command and Control 

46<Vb (16) 

3% (1) 

9% (3) 

11% (4) 

31% (11) 

35 

Total Respondents 

35 

(skipped this question) 

3 


6. During the course(s), which learning context did you like the most? 



Response 

Percent 

Response 

Total 

Interaction - exchange with 
fellow students 


71.4% 

25 


Interaction - exchange with 
instructor 


8.6% 

3 

Working alone 

—— 

20% 

7 

Total Respondents 

35 

(skipped this question) 

3 


*£ Internet 


Figure 30. Raw Survey Results Continued 
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Figure 31. Raw Survey Results Continued 



Figure 32. Raw Survey Results Continued 
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Figure 33. Raw Survey Results Continued 



Figure 34. Raw Survey Results Continued 
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Figure 35. Raw Survey Results Continued 



Figure 36. Raw Survey Results Continued 
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Figure 37. Raw Survey Results Continued 



Figure 38. 


Raw Survey Results Continued 
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V. DATA ANALYSIS 


Since this research is reliant upon a survey as the primary data collection tool and 
provides qualitative data by its nature, the main concern during analysis is addressing this 
potential limitations associated with self-report outlined in the data collection chapter. 
Essentially, the data analysis centers around building charts to reflect the data and then 
comparing those charts in accordance with the control measure described to minimize 
self-report. Because “hard” statistical conclusions are difficult to achieve with this type 
of data and approach, this analysis focuses on how closely the gathered data correlates 
with expectation based upon the literature review and draws inference from there. One 
key point for this analysis was the reliance upon the mode as the primary means of 
comparison. The mode was chosen because as the answer selected most often for a 
respective question, it reflects the “most right” response across a group that serve in a 
variety of services and billets. Because the Hypothesis Curve is continuous, it was 
converted to a discrete graph in order to facilitate comparisons based upon the mode. 
Figure 39 shows the converted Hypothesis Curve. This chart was constructed by 
following the Hypothesis Curve; however, it was built to reflect a single course time 
period. 


Course Specific Tacit Knowledge Base 



Time Periods 

Time Relative to Course 


□ Before 
■ During 

□ After 

□ One Year After 


Figure 39. 


Converted Hypothesis Curve 
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Respective charts were built in Excel using the modes as the data points for graphing. If 
the mode for an answer was in the N/A column, the answer with the next most responses 
for that question was used. If the data reflected a multi-modal response, the average of 
those modes was used for the chart. 

A. IS NPS DL COURSEWORK COMPARABLE TO RESIDENT 
COURSEWORK IN TERMS OF USABLE KNOWLEDGE GAINED? 

For this research question, analysis involved comparing each course individually 
to the Hypothesis Chart. The Excel CORREL( ) function was used to determine this 
correlation. The Primary Survey Question Set of questions 17-20 (Tacit Knowledge 
Base) was used for this comparison. As a means of cross-checking the data, the 
Associated Question Set of questions 8 and 10 was charted. 

1. Individual Course Charts (Tacit Knowledge Base) 

The individual class charts for the Primary Survey Question set of questions 17- 
20 (Tacit Knowledge Base) are as follows: 


Tacit Knowledge Base Reported for 103100 (Question 17) 


10-3100 Tacit Knowledge Base 


5- 0.8 


□ Before 
■ During 

□ After 



<u 
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□ One Year After 


£ 0.2 
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Time Periods 


Time Relative to Course 


Figure 40. 103100 Tacit Knowledge Base 


72 




























Tacit Knowledge Base Reported for SS3011 (Question 18) 


SS-3011 Tacit Knowledge Base 


0.8 



Time Periods 


Time Relative to Course 
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□ One Year After 


Figure 41. SS3011 Tacit Knowledge Base 


Tacit Knowledge Base Reported for IS3502 (Question 19) 


IS-3502 Tacit Knowledge Base 
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Figure 42. IS3502 Tacit Knowledge Base 
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Tacit Knowledge Base Reported for CC3000 (Question 20) 


CC-3000 Tacit Knowledge Base 
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Figure 43. CC3000 Tacit Knowledge Base 


2. Correlations Between Individual Course and Hypothesis 

Using the Excel CORREL() function, the correlations are as follows: 


103100 to Hypothesis Column Chart: 3)8 

SS3011 to Hypothesis Column Chart: .71 

IS3502 to Hypothesis Column Chart: .76 

CC3000 to Hypothesis Column Chart: /78 

3. Comparison to Support Tacit Knowledge Base Reports 

Question 8 and question 10 provided insight into the level of tacit knowledge held 
before and then after taking a DL course. Figure 6 depicts those two questions charted 
side-by-side for comparison purposes. Because there are only two data points in this 
chart, the Excel CORRL ( ) function cannot be used to make a correlation between this 
chart and the individual course charts. However, the Figure 6 clearly showed a higher 
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tacit base of knowledge after completing a course(s) for each course, which does follow 
the trend observed in the individual course charts. 

Chart Merging Question 10 (Before) and Question 8 (After) 

Before and After Confidence in Course Matter 

0.8 

S' 0.6- - - 

a) - 

o> 

1 04 -■- p—T- - 

5 

o 

£ 0.2 - --- 

o-LI- 

103100 SS3011 IS3502 CC3000 

Figure 44. Before and After Confidence in Course Material 

4. Observations 

The expectation was that the individual course charts would correlate relatively 
closely with the hypothesis chart. Because the hypothesis chart was a general case built 
upon literature review, a close correlation demonstrated usable knowledge gained and 
allowed for the inference of equality between resident and non-resident coursework on 
that metric. For this research, a relatively close correlation is defined as .80. The range of 
the correlations for individual classes is .71 to .98. The average correlation for all of the 
classes, which reflects the ISO program, is .808. The observable trend in Figure 6 
strengthened the argument because it clearly depicted a substantial increase of usable 
knowledge reported. 


□ Before 
■ After 


75 







































B. DOES NPS EFFECTIVELY AND QUICKLY DISTRIBUTE GRADUATE - 
LEVEL SKILLS VIA DL TO THE PERSONNEL SERVING IN THE 
OPERATIONAL BILLETS? 


For this research question, “effective and quickly distribute” means that usable 
knowledge was gained or refined. The assumption was that new knowledge gained or 
refined translates to new knowledge applied in the performance of billet responsibilities. 
In fact, this point is a significant potential advantage of DL coursework, because new 
knowledge can be applied as it is learned. The Primary Survey Question Set of questions 
7-11 (Billet Application) was used to cultivate insight into the reported application of 
skills to billet responsibilities. Question 9 and 11 were the primary questions for this 
research question. These questions queried the quality of the DL course(s) in providing 
applicable skills and use of those skills at work. Question 8 and 10 are focused more on 
tacit knowledge base and are not used in this research question’s analysis. The 
Associated Question Set of questions 12, 13, and 14 considered with question 7 queried 
to the application of DL coursework to billet performance in terms of frequency of use 
and direct application of course materials. The expectation is that questions 9 and 11 will 
correlate relatively closely suggesting that DL coursework provides applicable skills that 
were used in the performance of billet responsibilities. Additionally, the expectation is to 
see a relatively close correlation between the Primary and Associated Question Sets. 

1. Correlation Between DL Course Quality and Skill Use 

The correlation between the response averages for question 9 and 11 is .96. 

2. Comparison Between Frequency of Use and Course Materials 

Question 7 asked how much of the course(s) materials applied to current billet 

responsibilities. Questions 12, 13, and 14 asked how often you used new knowledge 
gained or refined from DL coursework in your billet responsibilities in terms of time 
relative to course completion. The modes for each class response to question 7 were in 
the 0-10% column, meaning that most respondents reported that ten percent or less of the 
course material applied to their current job. With one exception, 103100 after course 
completion, respondents reported their frequency of use of skills gained or refined 
through DL coursework at 0-10% before, during, and after course completion 
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respectively. These responses seemed to indicate a connection between course material 
that directly applies to billet responsibility and the frequency of use of skills gained or 
refined from that course. 

3. Comparison of Question Sets 

The Primary Question Set question 9 and 11 had categorical answers of Low, 
Medium Low, Medium, High Medium, and High. In order to form a comparison 
between the question sets, those categories required conversion to percentages. For 
comparison’s sake, Low became 0-20%, Medium Low became 20-40%, etc. After 
conversion, simple observation showed that there is very little correlation between the 
question sets. Almost all of the response modes for the Primary Question Set were in the 
60-80% or 80-100% categories. Conversely, all but one of the responses in the 
Associated Question Set was in the 0-10% category. 

4. Observations 

The expectation was that a relatively close correlation existed between the 
reported quality of DL coursework in providing applicable billet skills and use of new 
knowledge gained or refined through DL coursework in billet performance. A strong 
correlation between these two questions demonstrated that students felt like the course(s) 
are giving them applicable skills that they then use in the performance of their billets. 
This goes directly to the heart of this research question and gave evidence that NPS does 
effectively and quickly distribute graduate - level skills via DL to the personnel serving 
in the operational billets. Since the correlation between these two questions’ response 
averages was .96, strong evidence exists that NPS DL is succeeding in this mission. 
While a tight relationship existed among the questions in the Associated Question Set, the 
relationship between the two question sets was not closely correlated at all. The 
expectation was to see a relatively close correlation between these two question sets. The 
responses in question 9 and 11 clearly demonstrated that students feel they are receiving 
quality knowledge and reported using that knowledge 60-100% of the time in billet 
performance. A possible reason for the lack of agreement is that the respondents see a 
difference between knowledge (Primary Question Set) and skills (Associated Question 
Set). 
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C. IS THERE ANY TRANSFER OF KNOWLEDGE FROM A DL STUDENT 
TO THEIR ORGANIZATION? 


While the bulk of this research focuses on the usable knowledge gained by 
personnel, this research question seeks insight into how much of that new knowledge gets 
left behind in the organization after the DL student rotated to another billet. The 
foundation of this question was based upon the level of codified knowledge that remains 
in a work section for future personnel to employ. The Primary Question Set of questions 
21 and 22 (Transfer of Knowledge to the Organization) queried to the amount of new 
knowledge gained or refined through DL coursework that is incorporated into OJT, 
SOPs, procedures, etc. within the student’s work section before and after taking a DL 
course(s) respectively. The Associated Question Set of questions 7, 12, 13, and 14 
queried frequency of use of specific skills gained or refined through DL in the work 
section and course material applicability to the work section. The expectation is that a 
transfer of knowledge occurs between DL students and their work section. Part of this 
expectation is based upon the environment of being able to use some skills while learning 
them. The amount of transfer is expected to be somewhat low because the questions 
specifically primarily seek codified knowledge; however, some aspects of the questions 
such as OJT and informal training open the question to person-to-person knowledge 
transfer. Either way, tacit to explicit knowledge transfer is expected to be relatively low 
in this research because these types of transfers, Extemalization and Socialization, 
require thoughtful interaction and discourse, which may or may not be part of a military 
work environment that is mission-oriented [Nonaka 2003]. Additionally, the personnel 
turnover rate inherent to a military work section restricts long-term interaction between 
personnel. The expectation is that the Associated Question Set will support the Primary 
Question Set because the skills used in the work section will most likely be the ones 
codified for the future. 

1. Direct Query of Knowledge Transfer to the Organization 

Question 21 asked how much of the new knowledge gained or refined through 
DL was codified in the workplace while the student was taking the DL course(s), and 
question 22 asked the same question except the time frame was changed to after 
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completion of the course. In both cases, for each course, the reported percentage of 
transfer was 0-10%. While this result was low, it did relatively match the expectation. 

2. Use of Skills Compared to Transfer of Knowledge 

Question 7 asks how much of the course(s) materials applied to current billet 
responsibilities. Questions 12, 13, and 14 ask how often you used new knowledge gained 
or refined from DL coursework in your billet responsibilities in terms of time relative to 
course completion. The modes for each class response to question 7 were in the 0-10% 
column, meaning that most respondents reported that ten percent or less of the course 
material applied to their current job. With one exception, 103100 after course 
completion, respondents reported their frequency of use of skills gained or refined 
through DL coursework at 0-10% before, during, and after course completion 
respectively. 

3. Observations 

This very close match between the Primary Question Set and the Associated 
Question Set provided evidence that there was a transfer of knowledge to the 
organization and that it was proportional to the frequency of use of specific skills gained 
or refined through DL. 

D. CAN FREQUENCY OF USE OF SKILLS LEARNED THROUGH DL BE 

USED AS A MEASURE OF EFFECTIVENESS FOR THE DL PROGRAM 

AT NPS? 


This research question was focused towards further research. It attempted to 
evaluate frequency of use as potential metric for measuring the effectiveness of DL 
coursework in delivering usable knowledge. The genesis for this metric was the 
assumption that usable knowledge will be used. Identifying a concrete measure of 
effectiveness provides a valuable tool for analysis in the future. The Primary Question 
Set of questions 12 through 14 (Usable Knowledge Gained or Refined) queried directly 
to the frequency of use of skills gained or refined through DL course(s) time relative to 
the course. The Associated Question Set of questions 9 and 11 queried the quality of the 
DL course(s) in providing applicable skills and use of knowledge gained or refined at 
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work. The expectation was a high correlation between the question sets. A related 
expectation was that frequency of use would increase time relative to the start of the 
course, i.e., the farther along in a course a student was, the more knowledge in use in the 
workplace. 

1. Direct Query of Frequency of Use 

The Primary Question Set of questions 12, 13, and 14 queried the frequency of 
use of skills gained or refined through DL coursework before, during, and after taking a 
DL course. All but one of the response modes for each class in each time relative period 
was 0-10%. However, the modes’ values grew successively smaller from the before to 
during to after question. The degree of that shift was determined by computing a 
weighted averages from the before, during, and after responses. The weighted average 
method applied by the survey tool is as follows: 

• assign a multiplication value of 1 for the leftmost column of answers and 
increments by 1 until the rightmost column is reached 

• multiply the value by the number of responses in that column 

• add the columns and divide by the number of columns 

The weighted averages for the Primary Question Set are as follows: 

• Before (question 12) 

o 103100: 11.7 

o SS3011: 7.5 

o IS3502: 11.3 

o CC3000: 7.6 

• During (question 13) 

o 103100: 11.7 

o SS3011: 7.8 

o IS3502: 11.1 

o CC3000: 7.3 

• After (question 14) 

o 103100: 13.6 
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o SS3011: 
o IS3502: 
o CC3000: 


9.9 

12.8 

7.7 


2. Comparison of Frequency of Use of Skills and of Knowledge 

The responses in the Associated Question Set of questions 9 and 11 demonstrated 
that the students believe they received usable knowledge from their DL course(s). The 
survey tool generated a response average for these two questions. Question 9 asked for 
the student to rate the quality of the DL course(s) in terms of providing applicable skills 
for their billet. The range of the response averages was 2.2 to 2.85, and the average of 
the response averages was 2.55, which translates to the 51 st percentile. Question 11 
asked for the student to rate their use of knowledge gained or refined from DL courses at 
work. The range of the response averages was 2.44 to 3.19, and the average of the 
response averages was 2.76, which translates to the 55 th percentile 

3. Observations 

The mode for the reported frequency of use of skills gained or refined through DL 
remained in 0-10% range for almost all of the courses for the before, during, and after 
time periods. However, there was some reported increase in skill use for the “after” time 
period reflected in the weighted averages, which was more pronounced in the two more 
technical courses. Interestingly, question 11 which states “Rate your use of knowledge 
gained or refined from DL course(s) at work” resulted in an average response that 
translated to a 55 th percentile. The difference between the results for this question and 
questions 12, 13, and 14, i.e., 55% compared to 0-10%, for basically the same question 
except for the use of “knowledge” in question 11 instead of “skills” in question 12, 13, 
and 14 again leads to the possible distinction between these two terms for the students. 

E. INTERESTING POINTS 


The survey generated a couple of interesting points that are potentially significant 
for further research. 
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• Question 6 reported a 68.8% response choosing the interaction and exchange with 
fellow students as the preferred learning context for these DL students. Even 
more interesting, interaction and exchange with the instructor received a 9.4% 
response with the remaining balance of 21.9% of the responses preferring the 
context of working alone. The cumulative response for interaction and exchange 
between other students and the instructor was 78.2%. The reason this is 
interesting is that traditional thinking cites the lack of social interaction among 
students and staff as a significant hurdle for the DL learning modality in terms of 
delivering the same quality of education as resident coursework [Branstetter 2002 
p.76]. This response provided evidence to counter that assertion. 

• There is almost a 100% correlation between the modes for questions 7, 12, 13, 14, 
21, and 22, which suggested a linkage between course material that applies to 
current billet, frequency of use of specific skills gained or refined in a DL course, 
and the percentage of that knowledge that becomes embedded into the 
organization. 

• Students seemed to perceive a difference between “use of knowledge gained or 
refined” and “use of skills gained or refined”. 


F. FOLLOW-UP INTERVIEWS 


Because a difference existed between the expected relationship and reported 
results on questions that examined the use of knowledge gained or refined as compared 
with skills gained or refined in the workplace, a follow-up interview was conducted to 
gain insight into what caused that difference. Additionally, the follow-up interview was 
designed to get a little more demographic information as well as insight into the 
education versus skills question. The demographic information on rank, service, and 
billet was included to develop any relationship between the specific skills required for a 
billet and the skills gained or refined through an individual DL course(s). Essentially, 
one course in the certificate program may provide many skills directly related to billet 
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performance while the other courses provide a more theoretical or educational benefit 
that was not as easily quantified by the DL student. 

1. Follow-up Interview Questions 

The questions were as follows: 

• What was your rank when you took the DL courses? 

• What was your service? 

• What is your current billet? 

• Did you take all four courses in the Certificate Program? 

• Do you perceive a difference between use of graduate knowledge and use of 
graduate skills? If so, please give a brief explanation of that difference. 

• Have skills from a specific course(s) applied directly to your job much more than 
skill sets covered in other courses? 

• Do you see an indirect positive effect at work from graduate classes that may not 
have a large amount of directly applied skills for your current billet? 

2. Follow-up Interview Results 

On April 13, 2006, a follow-up interview was conducted via telephone with a 
Lieutenant in the U.S. Navy that had completed all four of the courses in the ISO 
Certificate Program. His billet at the time of the interview was a Maintenance 
Management Control Officer. His primary Designator was as an Aerospace Maintenance 
Duty Officer, but he wants to make a lateral move into the Information Professional 
specialty. This interview’s results are as follows: 

• Do you perceive a difference between use of graduate knowledge and use of 
graduate skills? If so, please give a brief explanation of that difference. 

Yes, the difference is related to Military Occupation Specialty and to 
personal goals. Knowledge can be applied in a much broader context than 
graduate skills. Graduate knowledge can help develop a better 
understanding of the big picture and help prepare for pursuit of future 
personal goals. For example, the ISO Program helped me have a better 
understanding of NMCI. It also makes me more competitive for 
promotion or a lateral move into the Information Professional community. 

Use of graduate skills really depends upon your billet. For instance, I see 
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more use for specific skills during the pre-deployment workup cycle 
[Toribio 2006]. 

• Have skills from a specific course(s) applied directly to your job much more than 
skill sets covered in other courses? 

Yes, SS3011 and 103100 provided specific skills I can apply during the 
pre-deployment workup cycle. I can see how IS3502 provides specific 
skills that I may be able to use dealing with the NMCI in the future 
[Torbio 2006], 

• Do you see an indirect positive effect at work from graduate classes that may not 
have a large amount of directly applied skills for your current billet? 

Yes, especially if you have an interest in that subject area. Because 
education makes you more competitive for promotion and other 
opportunities, you are more motivated to continue to excel in your job 
[Torbio 2006], 

On April 18, 2006, a second follow-up interview was conducted via email 
with Edward Basquill, a City Engineer for the City of Radcliff, Kentucky that had 
completed all three of the courses in the ISO Certificate Program. He was a 
LCDR in the USNR when he started the program and served as an IAP in that 
capacity. This interview’s results are as follows: 


• Do you perceive a difference between use of graduate knowledge and use of 
graduate skills? If so, please give a brief explanation of that difference. 

Skills are the ability to perform, whereas education is accumulated 
knowledge. Example: Information Operations provided a case study on 
operation overload, which lends itself to the skill of conducting an IO, 
whereas the Sun Tzu stuff was more literary, educational, or philosophical 
[Basquill 2006]. 

• Have skills from a specific course(s) applied directly to your job much more than 
skill sets covered in other courses? 

Yes. Although I am a civilian, the IO class relates well to marketing and 
dealing with the media in a technical job [Basquill 2006]. 

• Do you see an indirect positive effect at work from graduate classes that may not 
have a large amount of directly applied skills for your current billet? 
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Yes. For example, I applied some of what I learned in how I designed a 
public outreach website as part of my job for the environment [Basquill 
2006 ], 
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VI. CONCLUSION 


Conclusions for this research were developed through discussion of the survey 
results in the context of individual research questions. Establishing conclusions for this 
research provided the opportunity to recommend possible methods assessing and 
expanding the NPS DL program’s effectiveness in transferring usable knowledge quickly 
to military operations. A more robust capture of demographic information makes sense 
for future research to maximize analysis and results. 

A. DISCUSSION OF RESULTS 

1. Is NPS DL Coursework Comparable to Resident Coursework in 
Terms of Usable Knowledge Gained? 

The observations made in the Data Analysis chapter for this research question 
present a .81 correlation between the Hypothesis Curve and the ISO certificate program 
as derived from the individual courses in terms of usable knowledge gained. This 
correlation, which was based upon a general case built from literature review, verified 
that students gained usable knowledge from the DL coursework in the ISO certificate 
program. Drawing a comparison between DL and resident coursework at NPS required 
finding a measure of resident coursework’s ability to provide usable knowledge. As part 
of his December 2002 NPS thesis, Terry Branstetter measured the usefulness of the 
fourteen educational topics in the Information Systems Technology (1ST) in the 
fulfillment of Military Occupational Specialty (MOS) 9648 billet responsibilities. His 
research was conducted using Marine Corps Officers that had completed the 1ST 
curriculum at NPS and were serving in a tour utilizing that education or had completed 
that “payback” tour. Ligure 45 captures the educational usefulness of the topics in that 
curriculum and usefulness was described as topics useful to fulfillment of billet 
responsibilities. 
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Figure 45. Assessment of Education Usefulness (From Branstetter 2002) 

Branstetter further developed this point by polling personnel that were no longer 
serving in a “payback” capacity. Those results are captured in Figure 46. 


Respondent Assessment of Education Usefulness 
by Topic in Non-9648 MOS Billets(n=280) 



Figure 46. Education Usefulness in General Billets (From Branstetter 2002) 
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This is an important distinction because the educational usefulness reported in 
Figure 46 is in billets that are not specifically designed to leverage the graduate skills 
learned at NPS. In fact, the normal rotation from a USMC “payback” tour is into the 
operating forces. Since the student population of interest for this DL research is serving 
in operational billets, Figure 46 provided better data for comparison for this research 
question than the data presented in Figure 45. Figure 45 is included to provide a 
comparison for the two population characteristics. The survey for this research posed a 
question that asked the respondents to rate their use of knowledge gained or refined from 
DL coursework at work in question 11. With the assertion that using knowledge at work 
is equivalent to rating the educational usefulness of specific 1ST topics in billet 
performance, a comparison of question ll’s results to the data presented in Figure 45 and 
46 from Branststter’s thesis provided an almost direct comparison. The average response 
average for question 11 was 2.76, which translates to an average response of the 55.2 %. 
The DL survey reported response of 55.2% compared closely with the 57% reported in 
Figure 45 and the 46% reported in Figure 46. Clearly, DL coursework is comparable to 
resident coursework in providing useable knowledge gained. 

2. Does NPS Effectively and Quickly Distribute Graduate-Level Skills 
via DL to the Personnel Serving in Operational Billets? 

Question 9 focused on the quality of the DL course(s) in terms of providing 
applicable skills for current billet. The response average for this question was 2.55, 
which translated to 51%. This result was relatively close to the Branstetter findings 
presented in Figure 46, which was conducted on a similar demographic, i.e., personnel 
serving in operational billets. Additionally, a .96 correlation between question 9 and 
question 11, which asked the respondents to rate their use of knowledge gained or refined 
from DL coursework at work, further solidified the result form question 9. Coupled with 
the fact that DL students are learning and doing at the same time, the response from 
question 9, which is buoyed by the high correlation with a related question in the same 
survey and a strong supporting relationship to results from the Branstetter thesis, NPS 
does effectively and quickly distribute graduate-level skills via DL to personnel serving 
in operational billets. 
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3. 


Is There any Transfer of Knowledge from a DL Student to their 
Organization? 

The responses for questions 21 and 22, which directly queried to transfer of 
knowledge form the individual to the organization, conclusively show that a transfer of 
knowledge to the organization did occur. The observations from the Data Analysis 
chapter for this research question proposed a proportional linkage between the amount of 
knowledge transferred and the frequency of use of specific skills gained or refined 
through DL coursework in the workplace. Beyond that apparent linkage, question 7, 
which asked how much of the course material applied directly applied to their current 
billet, appeared to be proportionally linked to the other two responses as well. This 
relationship supported the point developed as an interesting point in the Data Analysis 
chapter that involved the linkage between course material that applied to current billet, 
frequency of use of specific skills gained or refined in a DL course, and the percentage of 
that knowledge that became embedded in the organization. The ultimate relationship 
between course materials that directly applied to current billet and the amount of 
knowledge embedded into the organization was not strong enough to suggest a causal 
relationship. The data provided enough support to conclude that course material that 
directly applied as an usable skill in the workplace was a related to the frequency of use 
of specific skills in the workplace and frequency of use of specific skills in the workplace 
was related to the amount of knowledge that was embedded into the organization. 

4. Can Frequency of Use of Skills learned Through DL be Used as a 
Measure of Effectiveness for the DL Program? 

Frequency of use was shown to be a reliable measure of effectiveness in resident 
coursework at NPS in the Branstetter thesis. Figure 47 captures the reported frequency of 
use of skills learned in the 1ST curriculum by Marine Corps Officers during their 
“payback” tour billets. Additionally, this data was similar in percentages reported to 
other data sets that presented responses capturing the value of theoretical knowledge and 
practical skills in the workplace. This data was used by Branstetter as a strong element of 
support for concluding that the NPS 1ST program significantly impacted Marine Corps 
9648 MOS billets [Branstetter 2002]. 


90 



Frequency of Education Usage 


70% 


60 % 


50 % 


% 4 ° % 

(/> 

§ 30 % 

Q. 


K 20% 


10% 


0% 


Figure 47. Frequency of Use (From Branstetter 2002) 






□ Kow ledge 

□ Skill 














□ 

1 - 1 


Daily Weekly Monthly Never 

Recurrence of Usage 


Acknowledging that frequency of use of skills was shown to be a measure of 
effectiveness in the Branstetter thesis, this research cannot conclude that frequency of use 
can be used as a measure of effectiveness for the DL program. The inconsistent 
responses between questions that queried to knowledge gained or refined through DL as 
compared to skills gained or refined as described in the Data Analysis chapter is the 
reason that no conclusion can be drawn. The suppositions that respondents perceived a 
distinction between knowledge and skill was one possible reason for this disconnect. In 
fact, the follow-up interview reported a distinction between knowledge and skills. 
Another possible reason for this disconnect is that perhaps the respondents do not 
consciously realize the amount of knowledge internalized and therefore cannot 
specifically differentiate between new knowledge and previous knowledge when it is 
applied in the workplace. Essentially, once knowledge becomes implicit, it is difficult to 
articulate. For example, in the follow-up interview, the Navy Lieutenant speculated that 
skills learned in IS3502 may help him in the future concerning NMCI integration in his 
Command; however, he reported the ISO program helped him have a better 
understanding of NMCI when he was differentiating between knowledge and skills 
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[Toribio 2006]. His responses suggest a perception that unless he was technically 
applying skills such as packet captures or IP addressing, then he may be using his new 
knowledge but not new skills. Those responses also suggested he did not perceive 
“having a better understanding of NMCI” as applying his new knowledge or skills. Only 
one follow-up interview was conducted because out of three known respondents 
contacted for follow-up interviews, only one respondent agreed to participate. Therefore, 
no conclusion can be drawn from the follow-up interview because it is only one data 
point. While the follow-up interviews were suggestive, more interviews are needed to 
verify the results from these two interviews. 

B. RECOMMENDATIONS TO NPS FOR DL 

1. Aggressively Promote Distributed Learning to the Services 

Distributed Learning provides tangible benefits to the Services because it delivers 
graduate-level education and skills to personnel while they are serving in operational 
billets. This is a clear advantage of DL over resident coursework. In fact, in many 
instances, resident students do not go straight to an operating force billet after graduating 
from NPS, so it may be as long as three years after a student graduates before they again 
serve in an operational billet. Beyond the immediacy provided by DL for the Services, 
this research demonstrated that DL is comparable to resident coursework in delivering 
graduate skills and education and that a percentage of that knowledge became embedded 
into techniques and procedures within the student’s workplace. From the follow-up 
interview, the notion of DL coursework making a person more competitive for promotion 
provides a key point as well. Personnel that are motivated enough to seek graduate-level 
education while serving in the operating forces are most likely focused on competing for 
promotion and challenging billets in the future, and the larger the group of motivated, 
talented people competing for promotion and command, the better for the organization. 
All of these points are definite selling points for the DL program, because they all 
directly benefit the Services. NPS should be looking to aggressively promote DL and 
expanding its DL program because of the service it provides the operating forces. 
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2. Solicit Input for Course Material from Commands 

Since there appeared to be a linkage between course material that applied directly 
to billet performance and the ultimate percentage of knowledge transferred to the 
organization, providing a method for Commands to request specific skills needed in billet 
performance allows NPS DL to accommodate those skills as much as possible. The 
benefit for this process is that the Commands become more involved and receive the 
benefit of having needed skills and knowledge transferring to their sections. Commands 
that feel involved in the process and satisfied in the resulting product will become 
advocates for the NPS DL program within the Services. 

3. Establish and Maintain Student Contact and Demographic 
Information 

A key challenge in fielding the survey was the lack of student contact or 
demographic information easily available at NPS. Student contact information was not 
tracked at all in Python and could not be produced. The only contact information 
available was through the Academic Advisor for the ISO Certificate Program. Those 
records consisted mainly of civilian email addresses and were heavily biased towards 
current students. The large number of “address not available” error message received 
when the survey was fielded confirmed this assertion. A central repository for current 
DL students in every curriculum provides a means for credible query across the entire 
program. Because the student population is largely active duty military, there is no 
reason that contact information cannot be maintained on a student as long as they remain 
in the active duty forces. Having reliable contact information allows for a continuous 
feedback mechanism for DL coursework and program management, which allows the DL 
program to constantly challenge and modify the courses in order to provide the most 
current, applicable product. 


C. OPPORTUNITY FOR FUTURE RESEARCH 


The primary opportunity for future research provided by this study is developing 
the frequency of use metric as a measure of effectiveness for DL. The survey design did 
not collect enough demographic information to make reasonable suppositions about why 
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a perceived difference between knowledge and skills existed among the survey 
population. Information about rank, service, billet, and billet responsibilities would have 
been extremely helpful in that regard. Even better than just a survey as a data collection 
tool would be a student logbook maintained throughout a course that noted the number 
of times specific skills learned in class were used in the workplace. Perhaps that could be 
a class project or offered for extra credit within the course(s) being studied. 

Another potential research opportunity would be to replicate a similar study but 
do it from the organization’s point of view. This would involve surveys and interviews 
with the student’s immediate supervisors. Data from this point of view would help 
solidify the conclusions made in this research and provide more insight into the benefit of 
DL for the organization. 
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